Big Data, Plainly Spoken (aka Numbers Rule Your World)

Yale meds, meet Yale stats

Over the weekend, Jason P. alerted me on Twitter to a new study from Yale that caught a lot of attention - because it's about scientists poking around our sewage. The initial touchpoint was about this chart:

The twitter user was annoyed with the double axes, of which he knows I'm not a fan. After I read the preprint, I was hoping someone might introduce these authors to the good people in Yale's statistics department.

The chart shown above is the centerpiece of the study. It shows on the left axis (red line) a measure of the viral load of SARS-Cov-2 found in human sewage, and on the right axis (black line) the number of new reported Covid-19 cases. Both metrics were measured for the New Haven metropolitan area, served by the sewage plant from which the samples were obtained.

The claim is that those two lines are highly correlated, with the red line leading the black line by roughly 7 days. The research team concluded that the viral load in sewage is a "leading indicator" of infections.

They then quantified this relationship by creating a piecewise linear regression model. As shown in charts D and E below, they fitted one line to the left of the peak and one line to the right of the peak.

***

These researchers are interested in a predictive model. If it is true that viral loads in sewage predicts the number of new cases seven days later, then they have themselves a predictive model. Unfortunately, the analytical methods used in this study fail to make this case.

A predictive model must predict the future, and not just explain the past. As I said before, some models explain the past well but that doesn't mean these models will predict the future accurately. Predictive models must be validated with data that haven't been used in building the model. This Yale study fitted a model to the entire time series (Mar 19 to Apr 30), which precluded any validation. That's the first problem.

Given their stated goal, the simplest model has the form:

New cases on day X = a + b * (viral load in sewage on day X-7)

Here, a and b are numbers learned from the training data. The "slope" of the regression line is b. For this model, if one wants to know what the new cases are for tomorrow (day X), one looks up the viral load from seven days ago, plugs that into the equation above, and out pops the prediction.

What is described in the preprint isn't operational as a predictive model.

As shown in the panel above, the researchers actually ran two regressions (D and E). As they explained it, D captures the slope as the cases ascend while E estimates the slope as the cases descend. This means that we must know where the peak is to even decide which model (D or E) should be applied. But the peak is not known until after the fact - so this model can't predict.

In addition, all time series in the analysis were smoothed using LOWESS. In other words, the number to plug into the above equation isn't the viral load from seven days ago; it's really the LOWESS-smoothed viral load from seven days ago. LOWESS smoothing averages values within a window centered at each day of the time series (it's a more advanced way of doing a moving average). This means the averaged value for today depends on values from the past as well as values from the future. A predictive model can't use data from the future. They need a smoother that only averages past values - but that might hurt their thesis because such smoothers are always lagging.

The same problem applies to the data series of new cases. Because of smoothing, the model has been trained to predict the smoothed value, not the value on day X. The smooth value is a weighted average of the values on day X, some days before day X and some days after day X. If you look at Figure 2C, you see the peak number of new cases to be just under 100. The following chart shows the raw data:

There were at least nine days on which the number of new cases exceeded 100. The effect of smoothing is to temper down sharp peaks.

And did they run the wrong regressions? I'm not sure since the actual equations were not included in the preprint. What is telling is their description of the slope as "1,305 virus RNA copies/mL per new COVID-19 case" (for the ascending model). Also, in visualizing the regression line, it's conventional to put the target variable (Y) on the vertical axis and the predictor on the horizontal axis. In the simple model formula stated above, the slope b is typically described as the number of new COVID-19 cases per unit increase in virus RNA copies/mL. In other words, it's the reciprocal of their description. 

It's odd to describe the slope as viral load per new case because the point of this predictive model is to use viral load to predict new cases. Recall that viral load is time shifted back by seven days. So, the slope as they described it is viral load seven days ago per new case today. It just doesn't sound right.

As I explained in my article for DataJournalism.com (here), regressing Y on X and regressing X on Y are two different things. The slope of one is not the reciprocal of the slope of the other. That's why we don't put the target variable in the horizontal axis, and the predictor in the vertical axis. It visualizes the wrong slope.

***
What is presented in the Yale preprint is at best an explanatory model. A common mistake is to assume that models that explain the past well will also predict the future accurately. See this post to learn more about why those types of models are different species.

To fix this preprint, they need to clarify what their model is, remove any dependence on future data not available to the model on the day of prediction, and include a validation study. A quick visit to Yale Statistics may prove very helpful.

 

The FDA looks past these statistical issues but maybe you shouldn't

Last week, I reviewed the explosive expose from Buzzfeed News on the Stanford antibody study. The first preprint from the research team was met with skepticism from numerous statisticians. Andrew as usual has a nice roundup of the key technical objections. Then, the researchers published a revised preprint that addressed some of the complaints.

Reading these preprints is a sobering experience. I talked to a friend who's in the business of developing tests, and apparently, FDA sets a low bar for tests like this, as it has little possibility of harm. In other words, they will look past most of the issues I cover in this post.

***

File Drawer Effect

The research team faced a difficult problem: outsider reviewers questioned their assumption of specificity central to their calculations. In particular, they know that the test's specificity (the ability to correctly identify negatives) has to be over 99 percent to defuse the statistical challenge.

To claim an extremely high level of specificity (i.e. almost zero false positives), the researchers  applied antibody testing to a sample of 30 blood specimens collected by a third party before the novel coronavirus was discovered, and reported that all 30 test results were negative. That was a woefully small sample to make such a strong statement, as I pointed out here. The Buzzfeed expose further revealed that a Stanford-affiliated lab conducted those 30 tests in an effort to validate the accuracy of the antibody test kit, which hasn't yet been approved by the FDA. The lab backed out of the study after seeing the validation results.

A back-of-the-envelope calculation indicates that at least 10 times the sample size is required to establish beyond reasonable doubt that the specificity is above 99 percent. Of course, the more negative blood specimens, the better. Procuring such samples is a bottleneck in the test development process. It's hard and expensive to acquire them. Labs usually purchase samples from other organizations, or they may contract other labs to conduct the validation testing.

In the revised preprint, the researchers now say they obtained 3,000 more negative blood specimens within just a few weeks (as I pointed out in the last post, it's never clearly stated whether they acquired blood samples, or just data). This effectively settles the sample-size objection. But the sudden emergence of thousands of validation samples raises an altogether different - and much thornier - question.

As one can imagine, the researchers spent the weeks in between preprints hunting for negative specimens. They knew exactly what they were looking for. They needed at least hundreds of specimens. The required sample size grew further if some of negative blood samples ended up with false positive results.

Statisticians hate getting caught up in this type of goal-seek dilemma. The situation can easily turn into a sequential optimization game: after each new acquisition, the researchers tally up the total number of specimens, and what proportion of the results are false positives. At each step, they decide whether to hunt for the next sample, or stop. (The third possibility is to reject the latest sample, and keep hunting.) In practice, one probably keeps hunting until a minimum sample size is reached, and the observed false-positive rate falls below 1 percent, or whatever the target level is.

Because the researchers can choose to stop at any step, and they can continue hunting till they achieve what they want, any result from this sequential process is over-optimistic. This is the exact same problem that plagues many A/B testing programs in the tech industry - the issue is known as "running tests to significance". Since these tests are never stopped when the results were insignificant (unless all patience has run out), but they are early-stepped at moments when the results showed significance, any conclusions are biased toward the positive.

If you're a sports fan, would you be delighted if I grant you the power to end a game whenever you like. What would you do? You'd never call an end if your team is behind but as soon as your team goes ahead, you call a stop. In that world, the winning percentage of your team is not an accurate reflection of your team's ability!

This statistical problem has various names, such as confirmation bias, publication bias, and file drawer effect. Andrew likes to call this "the garden of forking paths." Essentially, the researchers have too much control over the outcome.

The Stanford antibody study showed how such bias could infect a study quietly. We know that a second Stanford-affiliated lab performed a validation study of the test kit. We know that they compared the positives to a more established antibody test, and found roughly half the positives to be false positives. We know that this validation sample was not included even in the second preprint. (We learned of its existence thanks to the whistleblower report and Buzzfeed.) Its exclusion may have been a decision by the research team, or one made by the outside lab, as it also severed its tie with the antibody study. Regardless of who made the decision, the effect of its exclusion is publication bias. If unhelpful data are systematically vanished, the conclusion will be over-optimistic.

Selection bias

A quick look at the demographics table reveals that their sample of 3,000 residents was in no way representative of Santa Clara's population. It consisted primarily of white women. It had only 5 percent 65 and older. It had a third as many Hispanics as in the general population.

The only acknowledged recruitment channel was Facebook ads, which deploys an opaque targeting algorithm. My own experience with Facebook targeting gives me little confidence that it is "random" even when specified.

As a result of the obvious selection bias, the researchers re-weighted the data. This is a common procedure I explained previously in the context of exit polls (link). After this statistical adjustment, the reweighted sample had roughly the right gender, and race distributions. For example, in the raw data, whites had a weight of 64% and hispanics 8%; in the adjusted data, the weight of whites dropped to 35% while that of hispanics tripled to 25%. The true distribution of whites in the county was 33% and hispanics 26%.

The validity of this procedure rests on whether the 8% hispanics are a good representation of all hispanics, and so on for every subgroup they reweighted.

Lurking: age

Nowhere in the two preprints did the research team address the "age" problem. They included zip codes in their reweighting but not age. As a result, the age distribution even after adjustment deviated markedly from the county's true age distribution. Thirteen percent of the county's residents are 65 and over, a demographic group that is particularly badly hit by Covid-19. Yet, less than 5 percent of the adjusted sample are people 65 and over.

The research team said they did not use any other variables because doing so would create too many tiny subgroups, which is indeed not a good thing. I'd still like to hear why they thought the age adjustment is less critical than zip code adjustment.

Pooling of validation samples

It seemed that only 30 of the 3000 negative samples were tested at Stanford labs. All the other entries in the table of samples appear to be (a) manufacturer supplied data (b) data obtained by third parties or (c) test results performed by third parties on contract. I'm just guessing at part (c) as there was no indication they outsourced validation testing. It is possible that the extra thousands of negative specimens are all just data, without any new testing.

It's shocking to me that FDA may accept such data. It's also shocking that they allow the wanton pooling of arbitrary samples of data.

Here's how the specificity of 99.5% was ascertained. The researchers found from all corners (of the world?) 13 different samples of blood specimens ranging from 29 to 1,102 in size. Some of these are from the staff of the test kit manufacturer; some are "COVID-19-era PCR-negative specimens, for which some tested positive for another respiratory virus"; some are "pregnant women pre COVID-19"; etc. Basically, each sample has different characteristics, was collected for a different reason, and has different numbers of specimens. They are "pooled together" as if they are the same.

Why is this procedure suspect? Let's simplify and pool together two samples.

In this first scenario, you have two samples of 100 men randomly selected from the county. These two samples are statistically identical since they are random. By pooling them, you have total size of 200. They are still men from the same county. The two sample averages should be roughly similar as before pooling. The sampling error is reduced because of higher sample size. All good.

In the second scenario, you have one sample of 180 men selected from county A, and one sample of 20 women selected from county B. Using the same pooling strategy, you have total size of 200. After pooling, you ignore the gender or county of the people.

But it can't be true that the two scenarios can be analyzed in the same way, can it?

In the second scenario, the average has the implicit weighting of 9 men to 1 women, and 9 As to 1 B. Is your metric invariant with gender or residence? Probably not. So the average of the two samples is biased towards the average of men in county A. (Re-weighting is an attempt to correct such biases in the experimental sample but it wasn't applied to the validation samples.)

Meanwhile, the variability is underestimated. Variability is both between individuals, and between groups (gender and county). While the former is indeed reduced on pooling, variability due to gender and residence differences counteracts that. Underestimating variability means the margins of error are too narrow.

***

As mentioned at the outset, the FDA tends to look past these issues because they believe the potential harm of test kits is low. But these are practical examples of real-world statistical challenges, which I hope you find worth your time to know about.

 

 

Numbers indeed rule our world, 10 years out

Statistics is the subject of generalizing data. In statistics, we study how to use imperfect data to draw general conclusions. We often have to wait for the future to reveal the evidence of success.

Ten years ago, McGraw-Hill published my first book, a best-seller, Numbers Rule Your World (link). I’ve hoped to explain five of the most profound concepts from my field in a way that anyone can understand and apply to their daily lives. I attempted the balancing act of featuring specific stories while bringing out the general principles. In the last couple of months, the Covid-19 pandemic has generated immense attention to data and statistics, and provided me confirmation, ten years on, that the contents of Numbers Rule Your World (link) have general applicability. It affirms that statistics is a great subject!

If you’re approaching my book for the first time, or re-reading it, here’s how you can work your way through it, making connections at various points to the Covid-19 crisis.

***

Chapter 2: Bad Spinach / Bad Scores

This is your perfect starting point. Half of the chapter details a disease outbreak investigation. You learn about the information networks the CDC puts in place to monitor unusual patterns of disease around the country. You discover data analysts whose job is to set off the alarm. How do they decide if the first handful of cases foretell an epidemic?

You encounter “shoe leather,” the painstaking collection of data to pinpoint the origin of an outbreak. It’s much harder than you might think. The U.S. politicians are making much noise about the early days of Wuhan. An epidemiologist may learn that 10 out of 15 first patients all visited the Wuhan seafood market before they got sick. But the Wuhan market might be a hugely popular destination, like Central Park in New York City. Finding 10 out of 15 people went to Central Park over a weekend could be more common than you think. If all 10 touched a particular rare species of wildlife, that's another story.

Another controversy of the moment is the huge cost of shutting down the economy, against the benefit of saving lives. I invite readers to think about this cost-benefit tradeoff. Because the disease outbreak was traced to a spinach processing plant in California, the FDA ordered a recall of all spinach, which depressed sales of salad greens for six months or so. The cost was much lower than for this pandemic, but the bacteria is well-known and thus predictable, and the lives saved were most likely zero, and even harder to pin down than in the current crisis.

People are already complaining that social distancing and other measures have not improved our numbers, by saying deaths were “unexpectedly low”. You discover how hard it is to prove the effectiveness of a policy decision because you need the counterfactual – what would have happened if the FDA did not recall the spinach.

Chapter 4: Timid Testers / Magic Lassoes

Everything you should know about the statistics of diagnostic testing and predictive models in one place. The motivating examples are (a) the fight against performance enhancing drugs in sports via anti-doping tests and (b) the false promise of terrorism prediction algorithms.

I wrote several posts (1, 2, 3) about the flawed Stanford antibody study that reported 50 times more infections in Santa Clara county in California than reported cases. Despite this headline, the raw data is also consistent with a scenario of zero proportion having antibodies. The statistical reason is the low underlying prevalence of antibodies, which caps the number of true positives in any sample. The number of false positives will be large relative to true positives, because the false positive rate applies to those without antibodies, which is the bulk of the population.

An extreme case of this phenomenon is terrorist prediction using a dragnet surveillance dataset. The set of terrorists is a drop in the ocean of people under mass surveillance. For a predictive model to capture most of these terrorists (low false negative rate), it must flag a lot of people – even a tiny false positive rate when applied to a huge population results in a lot of false accusations.

You see the distinction between a “chemical” false negative (positive) and a “practical” false negative (positive). To evade steroids testing, dopers use all kinds of tactics, such as handing in someone else’s urine. The diagnostic test finds a true negative but in practical terms, it’s a false negative. For swab testing, one may test negative if the swab doesn’t reach a virus-rich area, or if the patient is tested at the “wrong” time.

All tests have errors. What are the consequences of errors? Also, how likely will the error be discovered? For anti-doping, do dopers inform the test lab they made a mistake?

Chapter 1: Fast Passes / Slow Merges

The first chapter gives important insights on managing capacity, which is a pivotal issue for hospitals during the coronavirus crisis. I cover two case studies: (a) Disney’s management of theme-park capacity, and (b) state’s management of highway congestion.

The strategy of building more capacity is unavailable during a crisis, and wasteful in general. In this extraordinary time, some localities were able to bring up temporary hospitals. Transportation experts discovered that policies of regulating congestion are more effective. Highway ramps slow down the influx of new vehicles onto the highway, and are turned on before congestion arrives. The key is to delay as much as possible the onset of congestion. Sounds familiar? It’s the analogue to “flattening the curve”. Social distancing is supposed to reduce contacts, slowing down community spread. This tactic delays the peak demand for hospital resources. The chapter explains the scientific support for why such tactics work.

Chapter 5: Jet Crashes / Jackpots

Statistical significance is the topic of this chapter, one of the most common and also most commonly mis-used concepts in statistics.

In the Stanford antibody study referenced above, statisticians argued that the observed level of positive results is consistent with zero prevalence. This is a way of saying the result is not statistically significant. What we mean is that the study’s result (50 positives out of over 3,000 tests) would be considered normal even if nobody in the population actually had antibodies. This conclusion does not assert that the true prevalence is zero. The topic of statistical significance is made inaccessible to non-mathematicians by textbook authors piling on equations, but it need not be so. I even made a video on explaining "not statisitical significant" (link).

Before you apply a test of statistical significance, you must make sure you’re comparing apples to apples. In the context of air safety, you shouldn’t compare U.S. and foreign airlines on all routes; you should only compare them on routes they share, in which case, you’ll find no statistical difference. One of the bad stats that is circulating during this pandemic is the purported harm of ventilators. This is usually stated as the vast majority of patients put on ventilators eventually die (presumably, relative to those patients not put on ventilators). But only the most serious patients are placed on ventilators. The proper analysis should focus on patients who qualify for ventilators, and compare those who were put on ventilators to those who got other treatments.

We have already seen a host of invalid comparisons. Comparing complete data from years ago with incomplete data this year. Comparing regions that are in different phases of the epidemiological timeline. Comparing countries that have taken different approaches to managing the crisis.

Chapter 3: Item Bank / Risk Pool

This chapter addresses a deeper issue in using statistical averages. An illustrative example is educational testing like the SAT. Critics pointed to the gap in the average scores across racial groups as evidence of bias. But the racial grouping masks a third factor, which is differential ability between students. Due to various socio-economic factors, white students have higher average ability than black students. So the real question is: Among those students with comparable abilities, does the test still show bias toward one racial group? (The answer is not straightforward.)

During the pandemic, the “denier” faction has argued that the mortality rate of Covid-19 is “on a par with” that of seasonal influenza. Are those two rates comparable? The simple answer is no. Comparing mortality rates masks a third factor, which is the level of immunity in the population. Even at the most optimistic, the current level of immunity to the novel coronavirus is less than 20 percent (This statement generously presumes that (a) having antibodies confer immunity without question and (b) the New York City test result is not a scam, and the test is 100% accurate). By contrast, because of vaccination and recurrence, a good chunk of people have immunity to influenza. The mortality rate of Covid-19 should be compared to that of influenza among a population that has no immunity to either, or a population with similar levels of immunity to each.

The other example in this chapter explains how the insurance industry is an outgrowth of the law of large numbers. In any given year, health or auto insurers cannot predict with high accuracy which covered individuals will file claims, but the magic of statistics allows them to predict with great accuracy the aggregate claims. Disaster insurers run into trouble when rare events happen, such as when Class 5 hurricanes hit Florida. Suddenly, potentially all covered individuals tap into the insurance pool all at once. Insurers try to pool together risks of different types and different areas. The global pandemic is a huge threat to the insurance industry. Perhaps there are lawyers reading this, and they might tell me there are natural disaster exclusions. If those don’t apply, we will see a gigantic number of covered entities filing claims all at once.

Florida’s hurricane insurance market was dysfunctional also because the risk is predictable along some parameters. Those living inland realized they were subsidizing the coastal dwellers and wanted out. The coronavirus also creates different risk subgroups. The younger generations have much lower risks than the elderly. When social-distancing and other practices are applied to the whole community, the subgroup with lower risk peels away. A key difference is community transmission – the risk of hurricane damages does not spread by contact.

***

The above covers less than half of the book. Almost everything in those pages has direct relevance to the Covid-19 world. It’s quite heartening to find out ten years later numbers indeed rule our world.

Here’s a link to get your copy of Numbers Rule Your World. There are also translated copies in Chinese (simplified and traditional), Japanese, Korean, and Portuguese, as far as I know.

Buzzkill: footnote to the Stanford antibody study

Credit: Kate Ter Haar/Flickr

Over the weekend, Buzzfeed News dropped a monumental bombshell, reporting on a whistleblower complaint filed with Stanford University about the infamous "Stanford study" - which signaled the start of the antibody testing movement in the United States. I previously wrote about that study here and here. I also explained why antibody testing isn't very useful.

The press coverage of the original result was distasteful from the start. The narrative pushed by the study's co-authors, hitherto respected scholars, was that the number of infections was "50 to 85 times higher" than the number of reported cases, thus the risk of death from Covid-19 was drastically lower than previously estimated, "about on a par with the flu", said one of the co-authors at a press conference. Some politicians and businesspeople immediately seized upon this finding to advance their "herd immunity" program, which is a nice way of saying let the old but not rich folks die.

Even beyond the questionable science of the study, the narrative was misguided. Little do people realize that only 50 people out of over 3,000 tested positive for antibodies in that experiment. Instead of reporting that only 1 to 3 percent of the participants tested positive for the antibodies, the journalists went with the researchers' framing of the number as being 50 to 85 times higher than reported cases. This is a common trick. It's like saying the 20% discount offer resulted in a 50-percent increase in sales, when sales went up from 4 to 6 while 100 sales are needed to break even on the cost of the offer. The result is statistically significant but practically meaningless.

Using this same tactic, one can say that the level of prevalence found in the Stanford study is at least 20 times below what is required to achieve so-called herd immunity. By the way, no scientist is willing to go on air to confirm that having these antibodies confers meaningful "immunity," which is saying a lot.

The argument for lowering the mortality rate is also overly simplistic. Epidemiologists have long distinguished between case mortality rates and infection mortality rates, with the understanding that cases are an undercount of infections. This study is an attempt to estimate the latter. Further, a higher proportion of infected has a couple of implications. First, the transmission rate is much higher, meaning that the reproduction number (the average number of people each infected person infects) must also be higher than previously believed, which ironically raises the bar for reaching herd immunity. Second, the gap between cases and infections reveals the failure of diagnostic testing, which suppresses both case counts and counts of deaths. Adjusting for cases but not for deaths clearly produces a mortality rate that is too low.

Furthermore, comparing the infection fatality rate of influenza to SARS-CoV-2 is apples to oranges. The infection of flu is dampened by the widespread use of the flu vaccine in the U.S. while we have no vaccine for coronavirus yet. This suggests that the base population upon which the fatality rate is applied is much higher for coronavirus so the same rate results in many more deaths.

***

The Buzzfeed investigation provides a lot of color to this antibody-testing movement. Here are the most important moments in the long report.

"I declined authorship on any manuscript... based on our testing" and "I ... end my involvement... without any ties, acknowledged or unacknowledged, to current, revised, or future contemplated preprints, publications or other public presentations of results" How two Stanford researchers disassociated themselves from the Stanford antibody study after performing validation studies of the test's accuracy.

Test kits come with claims of accuracy by their manufacturer. Accuracy is typically provided as sensitivity and specificity, measuring the test's ability to identify positives and negatives respectively. For the antibody test, given the low level of prevalence, the specificity has to be close to 100 percent, meaning all negatives must be correctly identified, for the test to have any utility. Otherwise, the high proportion of antibody-negative people in the population can generate more (false) positive results than the number of antibody-positive people who exist.

Vendors advertise test accuracy based on self-reported validation samples. Validation is hard - and for this reason, vendors can easily cheat to sell lemons. You can't validate the people who took your test because the point of taking your test is to determine who has antibodies. If you knew the answer, you wouldn't need to test them. So to validate an antibody test, you need to find blood samples that are unequivocably positive or negative. You have to purchase blood specimens from people who have them, or pay labs to conduct tests on your behalf on their specimens.

Or, you could take your blood samples, and apply a different test and see if the results match those from your own test.

The test kit used in the Stanford study has not yet been approved by the FDA so the need to validate the accuracy claims was even higher. The research team made at least two attempts to validate the test accuracy at Stanford-affiliated labs. Both groups later declined co-authorship of the study because they had reservations about the test results.

When the first preprint appeared, the specificity of the test was supported by a 30-specimen "independent" validation sample, plus manufacturer's data. The independent validation turned out to be the results from one of the labs that severed ties with the research project.

The murky disclosure surrounding the validation data

Many statisticians, including myself, raised concerns that the research outcome is highly sensitive to the assumption of the test's specificity. For validation, a sample size of 30 is way too small. I computed that a minimum of 300 - with all 300 testing negative - would be required to establish  specificity of 99 percent. In the revised preprint, the researchers expediently found 3000 more negative samples to justify an assumption of 99.5% specificity.

The authors said they "conducted additional testing to assess the kit performance, and continued collecting information from assessments of test performance to incorporate into the analysis." They "gathered all available information on test performance characeristics, with a focus on test specificity". In an addendeum, after mentioning the sample of size 30 referenced in the first report, they disclosed receiving additional information from "multiple sources, including the manufacturer's original data, test performance assessments for regulatory documents, and independent evaluations."

While reading all these text, I was scratching my head to figure out whether they conducted additional validation testing at Stanford labs, or they contracted other labs to perform validation testing, or they simply requested data from third parties who just happened to have them.

While they promised information on "provenance" of the data samples, one third of the 3,000 negative samples were simply described as "adults admitted to hospitals pre-COVID19." Which lab did the testing? When and where were these samples collected and for what purpose?

The ambient threat of confirmation bias

This search for samples creates a goal-seek situation for these researchers. They know what specificity is required to reach a scientifically acceptable conclusion. They have targets for how many samples are needed, and what the pooled accuracy rate has to be. We don't know how they conducted this search, and what measures were taken to prevent confirmation bias. If they had received data that took the accuracy rate below the required 99.5%, they could have continued searching for the next sample (or convinced each other that that sample was an outlier). I'm not saying they did this; the dangers affect anyone placed in this situation.

Well, well, well. There is a hint of impaired judgment. The researchers chose to publish their first preprint the day before receiving validation results from the second independent lab. By that time, the lab already expressed concern about false positives. The additional results came from re-testing the positives using ELISA, a "gold standard" test for antibodies. They showed that "a little over half" of those positives were confirmed. The just below half are false positives.

This validation sample showing low sensitivity was not included in the revised preprint while the other validation sample showing 100% specificity continued to feature. Both samples came from Stanford-affiliated labs that decided to decline authorship based on concerns about test accuracy. It's possible that the one lab did not authorize the use of their data. But this situation exactly illustrates the risk of "publication bias": whether the decision to exclude was made by the independent lab or the research team, the act of exclusion biased the statistical analysis.

Ironically, the Stanford team includes John Ioannidis, who has led an outcry against biased scientific findings.

"If you are willing to do a 5,000 test in New York, just tell me the cost and I will raise the money immediately." Jetblue CEO to Stanford researcher who was recruited to validate the test kit.

Jetblue's founder and CEO, David Neeleman, holds strong views on re-opening the economy based on the theory that a large number of people have been infected and are thus immune. It turned out he was a funder of the Stanford study.

The various co-authors of the study told Buzzfeed that they either had no knowledge of where the funding came from, or that there could have been multiple funders with different points of view (while maintaining they don't know who the funders are), or that their research could simply not be influenced by funding.

Neeleman now said that he was not shown results prior to the publication but in an op-ed 10 days before the publication, he wrote that the Stanford team "believe that the actual number of cases is very likely off by an order of magnitude of 10, or maybe even many times more". A few days before publication, he appeared on a Fox News show pitching a re-opening strategy, alongside with two of the researchers, who presumably still didn't know who funded the study.

One of the researchers suggested to Neeleman that he should "write a note [to the independent researcher] telling her you'll support her lab if she validates this kit." That's how the proposal for a large-scale New York test came into being. This researcher presumably didn't know Neeleman funded their study.

I noticed another discrepancy when it comes to funding sources: in the preprints, the researchers disclosed they "purchased" test kits from Premier Biotech but in one of the emails cited by Buzzfeed, the second researcher tasked with validating the test kit wrote "we are concerned about the specificity of the Premier Biotech devices that were donated for your study." The researcher could have been confused but if he was right, the "conflict of interest" section of the preprint should be corrected.

"If they had just done the New York study first, there wouldn’t be so much scrutiny." Jetblue CEO

Neeleman elaborated: "Unfortunately PR [public relations] impact and the ability to raise large amounts of money quickly will not be the same if you announce 1% of Santa Clara County tested positive for the antibodies versus 30% of New Yorkers which would be huge news." This email was dated before the publication of the first preprint, and way before New York State started antibody testing. Somehow, Neeleman knew (a) that the Santa Clara study would show about 1 percent positive even though the research team claimed he and no others were informed of results beforehand; and (b) that the New York proportion would be 15 times higher than reported (in the right ballpark).

Neeleman has been proven correct, as the New York result made quite a splash. As readers know, I have been complaining for weeks that New York Governor Cuomo - who is an avowed believer in science - has been promoting day after day after day results from an antibody testing program in his press briefings, while one quickly discovers that his Powerpoint slides are the sole source of data about this study.

Given what we now know about the Stanford study, I call once more for transparency. The State Department of Health, which is said to have conducted the antibody testing program, needs to come forward with experts, preprints and data. The citizens are entitled to know: who funded the study? What are the sensitivity and specificity claimed by the test designer? How did they validate their test kit? What analysis was done? What adjustments were made? They need to answer the same questions the Stanford team did.

The Times Union Albany reporters dared to ask questions, and everywhere they went, Department of Health officials claimed no knowledge of anything about this state-run program. I hope the New York Times is on this case.

What NY State did is much worse than the Stanford team. At least the Stanford researchers put information out there and responded to comments. New York has been feeding countless headlines, proclaiming 20 percent prevalence in NYC, when it has offered zero support for this.