Big Data, Plainly Spoken (aka Numbers Rule Your World)

Pre-processing data to avoid putting your foot in that puddle

Tracking apps are being touted as essential for the pandemic life. We are starting to see data analytics coming out of app developers (through their research partners). One of the earliest is the Covid Symptom Tracker app, developed by a team from King’s College (UK) and Zoe Global. I previously explained how they took such “big data” and drew conclusions about the general prevalence of SARS-CoV-2, and loss of smell and taste being the strongest predictor of an eventual positive test. (Link to the preprint)

(Here’s how I use the term “big data”. The OCCAM criteria are different from the usual interpretation of large volume, etc.)

In this post, I turn my attention to how the data was processed before entering the analyses. This is a particularly thorny problem for big data because the analysts exercise far less control over the data collection process than in more traditional studies.

In big data studies, one of the greatest challenges is sample bias. Sample size is usually of lesser concern. If the analyst wants to draw conclusions on the general population, the challenge is due to low or no representation of certain types of people in the sample. If the analyst intends to define new subgroups, there are no prior population studies to measure the bias.

In this post, I have divided the material into two sections. The first section discusses biases resulting from the pre-processing steps we've been told about. The second section speculates about potential biases, and asks for more disclosure on pre-processing.

Discussion of Disclosed Processing Steps

#1 Impossible values

Self-reported data are often filled with errors. In addition, bot traffic generates fake data. The Covid Symptom Tracking App is not immune to these issues. The researchers countered by filtering out impossible values. For example, body temperature was required to fall between 35 and 42 C. Weight must fall under 200 kg (440 lb). The maximum age allowed was 90, which is their most questionable decision. Covid-19 is the most deadly among those over 80. (That said, even if they allowed the oldest cohort, that group might not use mobile apps, which is still a bias.) What I’d like to see is reporting what percent of users were dropped because of these exclusions. Such accounting is key to judging the quality of data collected from apps. The preprint disclosed just one tidbit: over 80 percent of users were under 60.

#2 Missing values

In a second filtering, the Symptom Tracker team “selected 1,702 participants that reported (i) having had an RT-PCR COVID-19 test, (ii) having received the outcome of the test and (iii) symptoms including loss of smell and taste.” Item (iii) is difficult to fathom. It sounded like they retained only those users who answered the question about loss of smell and taste.

If this interpretation is correct, then they have flashed their hand – because they had a particular interest in loss of smell and taste (anosmia), they treated that symptom differently from other symptoms, thus introducing a confirmation bias.

Users who answered the anosmia question are more likely to have that symptom. Depending on the co-existence of anosmia and other symptoms, such filtering may have unintended consequences on the prevalence of the other symptoms in the analytical sample. There is no reason why anosmia should be treated differently from other symptoms.

A clear effect is that all symptoms have missing values except for loss of smell and taste. This creates the problem #3.

#3 Missing value imputation

In a subsequent step, the research team filled in the blanks. According to the preprint, “Prior to the modeling, to preserve the sample size, we imputed missing values for the symptoms of interest using missForest package in R”. So for any skipped question about symptoms, they used a model to guess what that user’s answer would have been, based on other answers of that user. Imputing missing values always introduce errors into the dataset; we do it because it’s the lesser evil. The problem is these errors only affect symptoms other than loss of smell and taste, because of #2.

I can’t say more about the impact of the imputation because the preprint did not show a comparison between the variables before and after imputation. What percent of the symptoms data were missing and subsequently filled in by this model, and how did the imputation change the statistics?

#4 People with no symptoms

In forming the analytical sample, the researchers also dropped all users who skipped all symptom questions. But not reporting symptoms might indicate no symptoms. Some users are too lazy to scroll through screens where every answer is No. Dropping all such users means the analytical sample is biased toward people with more symptoms, which is correlated with chance of infection. So the percentage testing positive in this sample will be higher than in the general population. (This is exacerbated by issue #5.)

#5 Triage testing

The other filter, those who have results from the PCR test, is another source of selection bias. That’s because the UK has been running “triage testing”: only those with severe symptoms, or have been in contact with infected people, or specific high-risk subgroups, are allowed to take tests. This too causes the set of app users not to mirror the general population. (The US, which is the only other country this app is currently available, also does triage testing.)

Since the tests missed all asymptomatic carriers, any analysis of test results will also miss all asymptomatic carriers.

Here is my Wired article about why triage testing creates problems for data analytics, and this Symptom Tracker App study is living proof.

 

***

 

The above discussion concerns pre-processing steps that were clearly outlined in the preprint. The next set of issues covers pre-processing for which no details were provided. I explain why how these steps may also introduce bias. It would be nice to know how the researchers handled these risks.

Further Questions about Pre-processing

#6 Levels of severity of each symptom

From one of the tables, I learned that the symptom of short breath has four responses: “short breath”, “short breath mild”, “short breath sign(sic)” and “short breath severe”. A fifth level of “no short breath” probably also exists. In the modeling, this information is magically collapsed into a Yes/No variable called shortness of breath. The same pre-processing affects most symptoms.

With five levels of short breath, the analyst has four places to plant the cutoff between Yes and No. What analysis was done to determine the cutoff?

The danger here is (unintentional) p-hacking. This is a technical issue which I won’t explain here. Just realize that the conclusion of the study may change depending on where those cutoffs are positioned. If you have 10 variables and 4 levels each, there are 4^10 = over a million (!!!) possible settings of these cutoffs. The analyst selected one of these settings.

#7 The time dimension

To complicate things even more, the Symptom Tracker app asks for daily reports. In one extreme, a user may have reported on shortness of breath for five consecutive days. Given 5 levels and 5 days, there are 5^5 = 3,125 unique sequences of {No, Severe, Severe, Mild, Mild}, etc. Which sequences are mapped to Yes and which to No? And why? If they went to the trouble of collecting the data in five levels, why not use them all?

The research team should provide data about the frequency of reports. Within the five-day analysis window, how many users submitted 0, 1, 2, 3, 4, 5, 6+ reports? When you ask this question, you might also ask about #8.

#8 Start dates

Compare two users who both submitted exactly one report during the five-day analytical window. One user downloaded the app on Day 1, submitted a report and was never seen again. The other user downloaded the app on Day 5, submitted a report, and has been reporting every day since. A proper study should establish a moving 5-day analytical window, adjusting for start date. Most big data studies I have seen ignore this bias.

#9 Users with intermittent missing values

Recall in #2, we learned that the analytical sample contained only those users who reported “symptoms including loss of smell and taste”. Does this mean the selected user must have reported symptoms at least once? Or every day s/he submitted a report? The problem for the analysts is that the act of not reporting symptoms may be highly correlated with not having the symptom at the moment.

#10 Dropouts, and inactivity

To add even more spice, the analyst must also deal with dropouts and inactive users. Some users may have downloaded the app and submitted a couple of days of data, then disappeared. If they didn’t delete the app, how did the analyst decide whether the users dropped out or were temporarily inactive? What were the reasons for suspending or stopping usage? Could it be related to recovering from Covid, or not experiencing symptoms? I'd like to see some statistics on dropouts and activity levels. Was the partial data retained, adjusted, or dropped?

#11 Mid-survey corrections

When the preprint said “the research team can add or modify questions in real-time to capture new data”, I was more troubled than reassured. How often was the survey changed? Assuming they did make at least one change, I wonder what adjustments were applied to the data to correct for it. These are tricky problems no analyst asked for. If an answer choice was removed mid-survey, what did the analysts do with the responses collected up to that point? If a new answer choice was added mid-stream, how did the analysts infer this variable for the period prior to the change? The analysts have two bad choices: deleting data, or filling in entire sections of missing values.

#12 Users who reported test results but not symptoms

Did some of the app users report test results but not any symptoms? Is this group retained in the analytical sample or removed? Based on how the pre-processing was worded in the preprint (Issue #2), it appeared that they were dropped from the analysis. Is it possible that this group are the elusive asymptomatic carriers? If the analysts kept this group, did they impute symptoms using the missing value imputation method (Issue #3). How accurate are these imputed values for this group?

***

Without having fuller details of the pre-processing, it’s hard to predict the direction or magnitude of biases introduced by specific steps. I hope you’ll appreciate how much massaging is required of “big data”.

In almost every case, leaving the raw data alone is like stepping into the big puddle of water right in front of you – not because you didn’t see it, but because you didn’t care about how deep the puddle is and what’s in the water. The wiser thing to do is to find a bypass. You may notice there are many other puddles all around you, but there might just be one less treacherous path.

***

I still have two loose ends to tie up in the near future. I’ll take another step back in the workflow, and examine the data collection through this Covid Symptom Tracker app. The other item is to return to the puzzle I included in the previous post on the methodology of the Symptom Tracker preprint.

 

[P.S. The blog on data collection is now posted.]

How to act like a data scientist 10: digging for information in unlikely places

Credit: RaymondClarkeImages/Flickr

I mentioned the other day that the testing data out of California looked weird (link).

On April 22, 2020, it released a gigantic pile of data, more than 165,000 results. This increased the total test results by more than 50 percent in one day. Most people kept moving along - telling themselves it's just a backlog being released.

If you're a good data scientist, you won't let this go so quickly. In fact, I won't hire anyone who accepts the backlog explanation at face value.

***

Tests are blood samples being processed at labs. Labs are currently swamped, therefore they are running at full capacity. This means a near constant rate of output. It's not that the labs suddenly found extra capacity to process 165,000 tests all at once. The normal rate of test results is in the 10-20,000 per day range.

So, after the labs completed the test analysis, the test results were held up somewhere. It's not a one day delay. It's about 10 days worth of results being locked up.

In addition, this batch of results are completely different than other days. From April 3-21, about 11 percent of all reported tests were positive. In the batch of 165,000 for April 22, only 1 percent were positive. This is an order of magnitude different, well outside random chance. For some reason, the test results that were delayed looked different from those that were released.

When I wrote the last post, I haven't looked at the test results released after April 22. On April 23, the proportion of positive results was.... you guessed it, back to 12 percent. On April 24, it was 16 percent.

Here's why the innocent explanation doesn't fly: let's assume the person who verified results before they were released disappeared for 10 days. So the results piled up on the person's desk, and s/he published them all at once. Those results should be even closer to the average proportion of positives because the sample size is 10 times bigger. Instead, the proportion positive is 10 times smaller!

As a data scientist, you've got to ask questions, and keep asking until you get a satisfactory answer.

***

What are some possible reasons?

Here's another plausible "innocent" reason. We might learn that the 165K results came from a separate, one-off initiative to get tests to a broader set of people including those without symptoms. Of course, we haven't heard of such a thing. So this is not accepted unless real evidence turns up.

Another innocent explanation might be that the California state contracted with additional labs to process the backlog. This can explain the 50% jump in test results but it fails to account for the extreme negative bias in the sample. The labs can't know which samples would come up negative before they processed them so for this reasoning to hold, the percent positive of those 165,000 had to be close to 11 percent.

Asking these questions led me to examine the daily testing report from California (link). I focus just on April because there was a reporting change in the first week of April; the numbers from March were small anyway. I saw a few other days in which 100% of test results were reported as positive (or negative). The chance of every single test coming up positive (or all negative) for any given day is practically zero. This means the release of the data is dependent on the test results. There is no scientific or statistical reason for this.

This is deeply concerning as a citizen. It makes one wonder if there are political considerations. As you could imagine, the tone of the media coverage can be swayed by selectively releasing positive (or negative) results. This conjecture too must be backed up with evidence.

***

I don't think we have an answer. Backlog is definitely not an acceptable answer. I encourage Californians to push your government to explain this. The questioning will be worth it. You will learn something about the testing program after this.

 

P.S. Thanks to the folks at Covid Tracking project who are keeping close tabs on the testing data.

The first major study using Covid tracking app data stumbles out of the gate

The tracking app studies are on rush order, just like all sorts of Covid-related preprints that have come under scrutiny. I learned about this collaboration between King’s College (UK) and an app developer Zoe Global through a news article proclaiming that 13 percent of the UK have already been infected with the novel coronavirus. This is the catch of the moment, being able to declare that you know what proportion of the population has already been infected.

I discussed the Stanford study the other day, which has received robust criticism from the statistical community (e.g. Gelman). This Covid Symptom Tracker study is much less convincing as a way to measure population prevalence. Its potential value is for rationing test kits to those most likely to test positive but notice that the objective of targeting the most afflicted conflicts with the objective of measuring prevalence, which is a feature of the general population.

***

Methodology

The UK team used an indirect path to estimating the prevalence of the SARS-CoV-2 virus. They launched a mobile app, which users downloaded and by which they submitted symptoms on a daily basis. A tiny subset of these users reported having taken a test, and submitted their test results. Using those users with known test results, the team then built predictive models relating the symptoms to the test results. They claimed that loss of smell and taste (anosmia) was the top predictor, even higher than fever or cough. They then used the model to score the large chunk of app users who submitted symptoms but not test results. The model predicted 13 percent of this group to test positive (should they get tested). The team then generalized this result to the general UK population. They (or the press) made a host of breathtaking claims: a) their predictive model can replace blood tests (“detect infections….without the need of extensive biospecimen testing”); b) the single predictor (anosmia) should be “used as a screening tool to help identify potential mild cases who could be instructed to self-isolate”; c) the 54,000 users predicted by the model to test positive “are likely to be infected by the virus”, which led to suggestions that official case counts drastically undercount infections.

Comparison to Stanford Study

This method of estimating population prevalence of the virus differs from the Stanford study in major ways. The Stanford study involved actual blood tests combined with survey data collected on the day of testing. The Symptom Tracker study collected survey data over time via an app interface, while the published analysis focused only on the first five days after the app launched. The Stanford study used actual results from antibody tests while the Symptom Tracker study asked users to report their test results from previously taken PCR (diagnostic) tests. The former looks for people who have recovered from a previous infection while the latter confirms people who were infected at the time they were tested.

***

I took plenty of notes while reading the preprint submitted to Nature (link). I grouped my comments into three buckets that reverse the analytical workflow. First, I discuss their conclusions while assuming that the analysis and data collection are accepted. Then, I point out major issues with the analytical framework. Last but not least, I raise concerns about how the data are collected and processed.

[P.S. This post became so long that I have to break it up into multiple parts.]

***

The fruit is dry, and the researchers kept squeezing

#1

Despite the headline number of 1.6 million downloads, 75% of those users never made it past the starting block. The data driving the predictive model contain about 1,700 users, those who reported both their symptoms and results from prior testing to the app. The model is then applied to another 412,000 users who reported symptoms but not test results. The authors referred these 412,000 people as a “test set”. This is an abuse of terminology. A test set must come with the answers. In this case, a proper test set should include the test results for every individual scored by the predictive model. The analyst can then compare the predicted result to the actual result in order to measure the model’s accuracy. No one knows a single test result for the 412,000 individuals in the “test set”.

#2

The oversight of #1 led the researchers to make the gravest error. They believed their own model. In stating their conclusion, they strayed from 13 percent predicted to test positive to 13 percent “likely to be infected by the virus”. This generalization presumes 100% accuracy for model prediction as well as 100% accuracy for testing.

#3

The press took the liberty to generalize further. Instead of 13 percent of users of the Covid Symptom Tracker app, they speak about 13 percent of the UK population. That step of logic requires that the app users are a random sample of the general public, which is obviously false. [I will address data collection and processing in a separate post.]

#4

We next look at accuracy metrics for the 1,700 users used to develop the predictive model. The reported sensitivity is 54% and the specificity is 86%. I’ll round these to 55% and 85% for our discussion. We are familiar with the concepts of false positives and false negatives because of Covid testing (if not, see my previous post). In the set of 1,700 users, the researchers said, one in three reported positive test results. So, the model’s job is to fish out the 567 positives from the pool of 1,700. Sensitivity of 55 percent means the model correctly predicted 312 of those 567 users. Specificity of 85 percent means the false positive rate is 15 percent, so out of the 1133 users with negative test results, the model incorrectly predicted 170 to be positive. Taken together, the model issued 312+170 = 482 positive predictions out of 1,700 users, roughly 28 percent.

The so-called positive predictive value for the predictive model is 312/482 = 65 percent. Out of every 100 people predicted positive by the model, we expect 65 to be correct while 35 are false positives.

From a predictive accuracy perspective, this is pretty good. If we grab a random sample of those 1,700 users, we will find a third of them to be positive (self-reporting) but if we select from the subset that are predicted positive, we will find almost two-thirds to be positive.

But the model (like any other) is nowhere close to perfect. Thirty-five percent of those predicted to test positive will test negative, if they take the test. The model also makes false negative errors. It flags 28 percent of the 1,700 as positives so even if every predicted positive is a true positive, the model cannot capture all self-reported positives, which is one-third of the group.

If you have the Numbersense book (link), you can review all of this material in Chapter 5, where I applied the same analysis to Target’s model for predicting which shopper might be pregnant.

#5

The percent truly positive is not the same as the percent predicted positive because of false positives and false negatives. A prediction cannot be equated to being “infected with the virus”.

The authors of the preprint did qualify their statement with “likely to be infected”. How likely is likely? Don’t hold your breath, the chance of testing positive can be just above 50/50.

To see this, you have to understand model scoring. The main model is a logistic regression, which means that each of the 412,000 individuals is given a score, to be interpreted as the predicted probability of testing positive for the coronavirus should they take the PCR test. The more relevant symptoms someone has, the higher the score. Since the score is a probability, its value ranges from 0% to 100%.

The analyst must translate these probabilities into a binary outcome (will the individual test positive or not?). This study did the most standard thing one can do - picking 50% as the cutoff score. All individuals scoring above 50 percent are predicted to test positive, and those scoring below 50 percent are predicted to test negative.

With a 50% cutoff probability, the marginal cases within the 13 percent predicted to test positive will have a chance of testing positive of around 50 percent. It’s quite, hmm, bold to say these people are “likely to test positive”.

#6

Now, let me explain why the percent predicted positive isn’t just somewhat useless; the percent predicted positive has nothing to say about population prevalence.

It turns out that for a given predictive model, the analyst decides what percent to predict positive. It’s not a passive outcome of the model. The analyst chooses its value by trading off false positives and false negatives.

If the cutoff is increased from 50% to 75%, then the proportion predicted positive will drop below 13 percent, but we will have more confidence that those predictions are correct – the people at the margins now have at least 75% chance of testing positive. On the other hand, we can expand the proportion of predicted positives above 13 percent, and then, we will be less confident of those predictions – some of the predicted positives will have less than 50 percent chance of testing positive. The point being that the modeler decides what proportion of predictions are positive, and thus, the methodology of this Symptom Tracker study can justify any level of “prevalence”. It’s really this bad.

#7

I previously told Wired one of the evils of “triage” or targeted testing is that it biases the samples available to build predictive models that are necessary to conduct triage. In other words, we’ve set up a vicious cycle. This predictive model is being proposed to implement triage testing – that is, let the model decide who should get tested. (It’s hard to lay bad ideas to rest! The other being “herd immunity”.)

We have a real-life demonstration of the problem in this preprint. As shown in #4, this predictive model guesses that 28 percent of the 1,700 users who had prior test results would have tested positive. Then, the model is used to score 412,000 users who have not taken the test. It guesses 13 percent of this group to test positive. Why not 28 percent?

If both groups of users have the same characteristics (say, both are random samples of the general population), then we should expect both proportions to be 28 percent.

However, in the UK, they severely ration testing, and only test people with severe symptoms. In Table 1 of the preprint, you can see that people in the 412,000 group reported lower incidence of almost all the symptoms listed, compared to the 1,700 who already were tested (even lower than the ones who tested negative).

Did the model realize that the untested people are less likely to test positive, and therefore predicted a lower proportion to test positive? I wish our models could be so smart. But no way! So why did the model flag only 13 percent of th 412,000?

I'm going to let you stew on this one a bit, and reveal the reason in my next post. It might help to use the Target pregnancy prediction model that I featured in the Numbersense book (link).

That model predicts the chance that a female shopper is pregnant because pregnant women spends money. Assume that I built the model using a list of women shoppers. The model makes predictions based on past purchases of certain products.

Now, I apply this model to score a list of male shoppers. This means I feed in data on whether these shoppers purchased those products included in the model. What scores (probabilities) will the men get? Does the model know the training data contain all women, and the "test set" contain all men? Please comment below if you need help thinking through this.

#8 (a minor point)

In the preprint, the 13 percent comes with a ludicrously tiny margin of error: [12.97%, 13.15%]. Typically, a small margin of error instills confidence that what was observed was unlikely to occur due to random chance. In this instance, the tiny margin of error is purely a result of throwing 412,000 names into the “test set”. When the sample size is so huge, the model almost always flags 13 percent as positive.

The uncertainty of the study’s conclusion arises from the modeling not the scoring of the model. So, if you want to understand the model’s accuracy, you have to go back to the discussion in #4. Those accuracy metrics have their own margin of errors, which are in the plus or minus 5 to 10 percent range.

***

What about the other big talking point coming out of this study - that loss of smell and taste is more important than even cough as a predictor of having SARS-CoV-2?

#9

Triage testing also dooms this finding (I know, it's sad to see). Because of pre-selection, the entire set of 1,700 people with test results have high incidence of all the symptoms. Thirty-three percent of them experienced loss of smell and taste. For persistent cough, it’s almost 50 percent.

Here’s the problem: if half the people have cough, cough isn't a good predictor of anything. Remember that only a third of this group self-reported positive test results. Any model that uses cough as a single indicator splits the group 50/50, which immediately leads to 28 false positives.

It’s no coincidence that 33 percent experienced loss of smell and taste, compared to 34 percent (to be exact) who self-reported positive test results. The reason why anosmia was the “best predictor” is because its incidence in the training data approximates the frequency of the positive outcome.

***

For data scientists and analysts, the big lesson is don’t feed biased data into model building.

 

[P.S. I have further comments on the data pre-processing and the data collection of the Covid Symptom Tracker study.]

Know your data 26: Confidence in numbers dropping

Now that you clicked on that, I'm here to tell you confidence - in - numbers dropping is not the same as confidence in numbers - dropping.

That's my reaction when I went to the Covid Tracking Project looking for California's testing numbers. Here's what I found:

Did someone explain what happened on Wednesday? The total number of test results jumped by 50% in one day, with most of those being negatives. The proportion of positive results went down from 11% to 7.5% in one day.

This type of weirdness makes data analysts nervous. It's hard to trust the data, and it destroys any ability to analyze trends.

To justify zero false-positive assumption, Stanford study needs a reference sample 10 times bigger

Credit: NIAID

Today, I return to one of the statistical issues that doomed the Stanford study that claimed the true infections are 50 to 85 times the reported cases. You can read the entire critique of that study here, or Andrew's more technical and longer debunking here.

Here's a summary: 97 percent of the residents of Santa Clara are believed to not have the antibodies for SARS-CoV-2, and so even a small false-positive rate when multiplied by so many people will produce a large number of false-positive test results. These false results vastly outnumber the true positive results since only 3 percent of the residents may have those antibodies.

Thus, the test needs to have a tiny false-positive rate, meaning it must have a high true-negative rate (technically called specificity).

The study's researchers estimated the true-negative rate by running the antibody test on 30 samples of blood collected prior to the advent of Covid-19. The test returned 30 negative results. Thus, they asserted that the specificity of the test is 100 percent.

Statisticians don't accept perfection. Even if you get 30 heads in 30 coin tosses, it is still possible that the coin might show a tail on the next toss. We consider what might happen if the specificity wasn't 100 percent but say, 98 percent. That 2 percent false positive rate turns out to make a mockery of the study's headline result.

Because almost all people do not have antibodies, a 2 percent false positive rate produces more false positive results than the 50 positive results observed in the 3,330 tests. So it is possible that all of the 50 results are spurious. Not a good look!

***

Today, I look at this same problem from a different angle. The researchers might want to stick to their assumption of specificity close to 100 percent. One way they can achieve this is by enlarging the reference sample used to establish the specificity.

It turns out that 30 samples are not big enough to be sure. To really be sure that the specificity is at least 99 percent, they need 300 samples to all test negative, 10 times more than they used.

***

Let me explain how I get to 300 samples.

First, assume the test has 100% specificity, that is to say, if the subject has no antibodies, the test will always come back negative. There is no chance of a false positive. In this case, if we apply to test to 300 pre-Covid blood samples, all 300 will definitely come back negative.

Now, let's say the test has 99% specificity. False positive is a possibility, albeit remote. The chance of 300 negatives in 300 samples is (0.99)^300 = 5%.

If the test has 98% specificity, the probability of 300 negatives is now (0.98)^300 = 0.2%. The lower the specificity, the lower the chance of all 300 samples coming back negative.

In statistics, we like to ignore "rare events". Rare events are usually defined as events that happen with less than 5 percent chance. We focus on things that happen 95% of the time.

So in the above scenarios, once specificity goes below 99%, the chance of seeing 300 negatives in 300 samples drops below 5%. Any event that happens with 5 percent chance or less is a "rare event" which we choose to ignore.

Thus, we say that when specificity is below 99%, there is essentially no chance of all 300 samples coming back negative. [ We say this with 95% confidence, meaning we allow ourselves a 5% margin of error. ]

In other words, if the researchers ran their antibody test on 300 pre-Covid samples, and got 300 negatives, then we can safely say that the specificity is above 99 percent.

***

How is 30 samples too small? We can go through the same analysis.

If specificity is 100 percent, then of course all 30 samples will come back negative.

How low can specificity go before the chance of all 30 negatives dips below 5 percent? Turns out this is 90 percent specificity. With 90%, the chance of 30 negatives in 30 pre-Covid samples is (0.9)^30 = 4%.

So even though the test was perfect for 30 samples, the chance of a false positive is not 0 percent; it could be as high as 10 percent. A 10-percent false-positive rate is devastating to the Stanford study. Out of 3,330 test-takers, we expect 3 percent to be infected, so 3,230 people have no antibodies. Ten percent of these would nonetheless get positive results - that would be 323 false positives. But they only reported 50 positives so quite possibly all of them are incorrect!

***

This is a great opportunity to discuss a common statistical problem. What can you say when nothing happens in your sample? For example, you flip a coin 10 times and you get all heads, or all tails (equivalently, no tails, or no heads).

In a nicer setting, we might get 4 heads of out 10 flips. Then, we say the coin has a 40 percent chance of showing heads. We can put a margin of error around that 40 percent number. (This is covered in every Stat 101 class.)

What if you got 0 heads out of 10 flips? Following the same method leads to the ridiculous conclusion that the coin has zero chance of showing heads (equivalently, 100 percent chance of showing tails.) How can you put a margin of error around that estimate of zero (or 100) percent?

Note that this is exactly what I did above. Instead of concluding that the specificity is 100 percent based on 30 samples, I'd say specificity is between 90 and 100 percent. With 300 samples, it's between 99 and 100 percent.

The logic I used above has been simplified to a "rule of three". Take 3 and divide by the sample size. For 30 samples, I have 3/30 = 0.1 or 10 percent. That's the size of the margin of error. So, for the all-heads case, the margin of error is 90% to 100% (width = 10%). For the zero-tails case, the margin of error is 0% to 10% (width = 10%). For 300 samples, the width is 3/300 = 1%.

 

 

Why tests are not perfect

From Andrew's blog, I learned about Tom Lumley's post that provides insight into the accuracy of the two types of Covid-19 tests. Tom is a biostatistician.

As you probably know by now, there is a diagnostic test, the most prominent of which is called the (RT-)PCR test, that involves using a swab to get sputum from the nose or throat. This material is taken to a lab which looks for the presence of SARS-CoV-2, the name for the virus that causes Covid-19 the disease. This test identifies who currently are infected.

Then, there is an antibody test, which looks for antibodies that are found in the blood of infected people, as the body fights against the virus. This test does not detect the presence of the virus; it looks for a sign that it passed through.

These two tests are not interchangeable. Some politicians on both sides of the Atlantic try to cover up the slow rollout of diagnostic testing by talking up antibody testing. In fact, early reports from antibody testing program such as the Stanford study found less than 5 percent have antibodies which means the virus is not as prevalent as claimed. (Here's why many statisticians are dismissing the Stanford study.)

It might seem like a simple issue: is the virus/antibody there or not? In both cases, if someone were not infected, their sputum or blood shouldn't have that specific virus or antibody in it. Why can't tests be perfect?

***

Let's first consider the antibody test. Antibodies are generated by our bodies naturally to fight an antigen. According to Tom, the antibody to SARS-CoV-2 is not unique - it varies from person to person slightly. This means that any test needs to look for a "range" of antibodies. To complicate matters, antibodies to similar viruses are also similar. Thus, the test will not find the exact target antibody but something similar. Then, the test has to decide whether the difference is due to different individuals or different (but similar) antigens.

One should hope that the person-to-person variability is smaller than the antigen-to-antigen variability. A test would then set up a "range" of outcomes that qualify a positive finding. If the measurement is near the edge of this range, there is a potential for error.

If the acceptance range is too "tight", the test might mistakenly decide it found antibodies for a different antigen when it's just a different individual. This produces false negative results. If acceptance is too "lax", it might mistakenly conclude that an antibody for a similar infection is an antibody for SAR-CoV-2 for a different individual. This produces false positive results.

There is a trade-off between these two errors, controlled by how the test developer sets the acceptance "range". For the antibody test, doctors don't like false positives. These are people who don't have antibodies but are told they do. As I discussed the other day, because the vast majority of people don't have antibodies, even a tiny false-positive rate will result in a lot of people being told they have antibodies when they don't.

The ability to correctly identify people without antibodies is called specificity. Some researchers are assuming antibody tests could be 100% specific. I think that is overly optimistic.

***

The PCR diagnostic test illustrates another aspect of testing accuracy. Sometimes, the error arises externally. In Chapter 4 of Numbers Rule Your World (link), I demonstrate this using steroids tests. A false-negative result arises when the lab mistakenly concludes the amount of chemicals in the doper's urine is not high enough to warrant a red flag. That's a "chemical" false negative.

But a false-negative result is really just any doper passing an anti-doping test. Someone might submit a fake urine sample, or hide from inspectors. In one case, the fake sample will not test positive, and in the other, there will not be a test. Both should be considered false negative cases even though the testing lab did not make a mistake.

Tom says "it is almost impossible to get a positive [PCR test] result without any virus". But low false-positives usually mean higher false-negative results. One reason for false negative is testing at the wrong time, or failing to swab properly. Bad timing is particularly important because "if you get tested too early, there might not be enough virus, and if you get tested too late the infection might have relocated to your chest."

Timing is also important for antibody testing because "a positive reaction [from one's body to the virus] takes time — at least a few days, maybe a week — and it stays around at least a short time after you recover."

Timing is an external factor. The consequent false negative error has nothing to do with the testing lab. Even if the test itself is perfect, there will still be people walking around with the false security of a negative test result.

So errors from testing occur because the test itself cannot be perfect (e.g. antibodies against similar infections may look like antibodies against the same virus coming from different individuals), and even when the test is close to perfect, other factors outside the lab, such as bad timing, cause errors.

***

One final comment on the politics. At various times, some officials argue that the rollout of diagnostic testing has been slow because tests are not accurate enough. Don't believe a word of that.

Let's say 5 percent of the population are infected. Borrowing some numbers from Tom, I assume that the diagnostic test has 35% false negative rate and 2% false positive rate. For every 1,000 (random) people given the test, there are 50 infected and 950 not infected. Of the 50 infected, 65% or 33 will get a positive finding and 17 will falsely believe they don't have the virus. Of the 950 not infected, 930 are told they don't have the virus correctly, with 20 told they have it when they don't. In total, 930+33 = 963 out of 1,000 will get the right answer.

So what the politicians are really saying when they reject this test is that having 17 out of 1,000 people falsely believing they don't have the virus thus spreading it is reason enough to deny 963 out of 1,000 knowing the correct answer. For every false positive, the test identifies two true positives, who can be isolated. It doesn't stem the spread completely but it does slow it down. Given that these politicians are all in for "flattening the curve", they should be clamouring for more testing.

 

 

 

 

 

 

The percent with antibodies could be 3 percent or it could be zero. That's what the Stanford study really said.

As predicted, test accuracy is getting major play in the media this week. One of the big headlines involved the claim that about 3 percent of the U.S. population may already have been infected and recovered, according to a study coming out of Stanford. The good thing about this study is its attempt to escape from the straitjacket of "triage" or targeted testing. It recruited about 3,330 people without requiring that they have symptoms. They were tested for antibodies, which is an indication that the person has recovered from a previous SARS-CoV-19 infection. Showing that a large proportion of the population have been infected and recovered is key to those pursuing the "herd immunity" theory.

The bad news is that the methodology of the study has already been debunked by Andrew Gelman. Andrew's post is here in its entirety. In this post, I give you an executive summary, especially for those not trained in the statistics of testing.

***

Before I get to the numbers, the idea that this study supports the "herd immunity" theory is far fetched. Herd immunity models typically show the vast majority of the population would be infected, much higher than five percent. We would need about 20 times the number in Santa Clara to get to herd immunity. Also, the presence of antibodies is not identical to having immunity. It is still not clear whether humans have immunity after recovery from Covid-19, and if so, for how long. The headline drawing attention to the framing as 50 to 85 times higher than reported is burying the fact that they found antibodies in only 3 percent of the samples.

If the result of this study is accepted, it exposes the sham of diagnostic testing in Santa Clara.

***

There are two major statistical issues with the Stanford study. Both are related to the construction of the sample - its composition and its size.

1. The way volunteers are recruited introduces bias into the statistics.

The 3 percent headline result is statistically adjusted. The 3,330 tests actually returned only 50 positives (1.5 percent). The adjustments were necessary because the researchers discovered biases in their sample by comparing basic demographic statistics to the county averages.

Bias means certain subgroups are under-represented in the sample. If a certain subgroup (say, men) are over-represented, then the composite subgroup (women) is under-represented so it's really two sides of the same coin.

While the sample size of 3,330 for one county is pretty large, the selection of volunteers was decidedly not random. People were recruited through Facebook ads, and they must drive themselves to the testing centers. 

A non-random selection has the chance of introducing unknown biases. In this case, some possible sources of biases are:

• people who have had difficulty getting tested (e.g. health workers) may show up in larger numbers
• some people who strictly adhere to the shelter-in-place order may not show up
• some people may like getting tested for free
• some people may believe Covid-19 is a hoax, or testing is pointless
• some people may find it more convenient to get to one of those test centers
• some people do not have Facebook
• some people do not respond to Facebook ads
• people who already have been tested, especially those who tested positive and recovered

These biases draw more or fewer infected people to the testing sites, and the strength of one versus another also matters. There is one bias that is irrefutable. Anyone who has severe symptoms or are hospitalized will not be represented at all.

It seems hard if impossible to measure and then correct for these biases, which is why statisticians usually recommend random selection. An ounce of prevention is worth a pound of cure. (A proverb that is most appropriate for this pandemic!)

It is sometimes mistakenly believed that larger samples cure sample bias. It doesn't. No matter how big the sample is, they will not find hospitalized patients in their sample. But those are almost surely infected.

***

2. The proportion of antibody-positive reported in the study is so small it may be all noise.

It turns out that even with 3,330 tests, a pretty large sample, the key result that 3 percent have antibodies could all be noise.

What is statistical noise? Imagine you repeat this experiment many times. For each set of 3,330 volunteers, you record the number of positive results. You aren't going to get 3 percent each time. The proportion with antibodies fluctuate with each sample but these variations are meaningless, an artifact caused by sampling.

The following analysis follows the plan used in Chapter 4 of Numbers Rule Your World (link) to analyze steroids testing and terrorist prediction systems. So if you have the book, you can compare the current situation with those stories.

In the raw data, the researchers found 50 positives out of 3,330 tests. (As mentioned in the first section, this 1.5 percent was revised up to 3 percent due to statistical adjustments which I won't describe here. If interested, Andrew has comments on those.)

Every test has errors. Some of the 50 positives are false positives. The false positive rate of a test describes the chance that the test will say someone has antibodies when they don't. The FP rate is one measure - but not the only one - of test accuracy.

To illustrate the issue, we consider the implausible, extreme scenario where none of the 3,330 volunteers have antibodies. So in a perfect test, the number of positives should be zero. If we get 50 positives, all 50 are false positives since no one has antibodies. That implies a false positive rate of 50/3330 = 1.5 percent.

Next, we assume that the researchers are right, that indeed 3 percent of the population has antibodies. (We further assume that the sampling is not biased.) That means the sample would be split 3% and 97%, that is, 100 people with antibodies and 3,230 without.

Let's proceed. We think that the test has a false positive rate of 1.5 percent. Applied to the 3,230 test-takers without antibodies, this antibody test should report 48 (false) positives. Note that in the actual experiment, the researchers counted 50 positives, so almost all of these are false positives!

Here is where things get confusing. Even though the test is rated as very accurate, 1.5 percent false positive rate, 98.5 percent true negative rate (also known as specificity), almost all the positives are false positives! The reason is the rarity of antibodies. The 1.5 percent false positive rate is applied on almost the entire test population because very few has antibodies. A small percentage of a large base is still a large number. (Another tie-in to the pandemic: a low fatality rate applied to a large infected base through fast community spread leads to a flood of deaths.)

***

When developing a new test, the false positive rate is not known because if we knew who has antibodies and who doesn't, we don't need to test people. We test because we don't know. In the above calculation, I used a thought experiment, the extreme case, to guess at the false positive rate.

The Stanford study had a better way of estimating it. They applied the antibody test to some blood samples collected before the Covid-19 pandemic when they're sure these should test negative if the test is accurate. Using that data, Andrew pegged the false positive rate at 0 to 2 percent.

At one end, the false positive rate may be 2 percent (instead of the 1.5 percent I used before). In that case, the test-takers without antibodies generate 65 false positives. This number is higher than the total number of positives (50) found in the experiment. So, test inaccuracy might account for all positive results!

At the other end, if the test was so accurate as to have zero false positives, then by definition, all 50 positives would be true positives, and we could run with their result. Anyone who knows a little about testing recognizes this scenario as implausible. It also implicates a high false negative rate!

Remember if the population were indeed split 3% with antibodies, 97% without as per the study's conclusion, there are 100 people with antibodies in the test sample. The test only returned 50 positives so the other 50 are false negatives.

This is the crux of understanding statistics of testing. You can't focus on just one accuracy metric. When one number goes up, another goes down. Here's the bottom line: if we think there are 100 people with antibodies in the test population of 3,330, then any reasonable test must return at least 100 positive results if it had any hope of catching all those with antibodies. That implies the test must have a positive rate (true or false positives included) of at least 100/3330 = 3 percent. To account for false positives, the test must return even more positive results. If it doesn't, then there must be false negatives.

False negatives are considered less damaging than false positives in the context of antibody testing because these people with antibodies would be taking unnecessary caution. It's not completely harmless since they might suffer from lost income. False positives are feared as they might have a false sense of security and take unknowing risks, and get infected.

Let me now relate this back to sample design. The above analysis shows that the observed test result is consistent with 0 percent of the population having antibodies. This is different from saying we believe the true proportion is 0 percent. It could be 3 percent as the researchers asserted. But the evidence is not strong enough to rule out 0 percent. In other words, even though 3,330 is a lot for one county, the sample is still not large enough to measure such a small signal.

***

I hope you're excited to learn more about testing.

See my previous post, or read Chapter 4 of Numbers Rule Your World (link).

 

 

 

 

 

 

 

 

The inutility of raw data 2

The last week saw more developments that show yet again why we need to statistically adjust data and throw away the raw data.

For some time now, we know that Covid-19 case counts have been artificially suppressed by restricted testing. The number of people infected with the novel coronavirus is vastly higher than the reported count. This leads people to conclude that the case-fatality rate (CFR) has been over-estimated because we should be dividing by a larger number of cases. So, the disease isn't as scary as once thought.

I pointed out that the news isn't all good. As the CFR falls, the rate of infection goes up. Both rates are affected by the number of reported cases. We have thus underestimated how widely the coronavirus has spread.

Last week, we received even more bad news. We learned that most (if not all) countries have under-reported the deaths due to Covid-19. (The Economist article linked to in my previous post is the best source for this.) Officials have only reported deaths from hospitals when many died at home or at care homes. Also, they require a positive test result to confirm a Covid-19-related death when few tests were conducted.

The adjustments to the fatality counts are huge, in the order of 50% to 100%.

Ouch! What does this do to the CFR? Double the deaths mean double the rate.

***

What might be the next shoe to drop?

Test accuracy.

On this, we have some initial reports of concern about "false negatives". For example, see this Bloomberg article.

I don't know that we have definitive knowledge yet of the accuracy statistics. But let's assume for the sake of argument that the COVID-19 diagnostic test has high false negative rates. This means a good number of people who are infected are mis-reported as not infected.

Correcting such a misclassification raises the number of infections as well as the number of deaths since both would be under-reported.

A double whammy! The rate of infection will have to be adjusted upward again. The CFR will also likely be adjusted upward.

***

Basically, any analysis that doesn't try to compensate for these inaccuracies is seriously flawed. Compensating for them requires making assumptions because we can't change the past. Since we didn't test those people, we would never know if they were positive.

With stay-at-home rules, a lot of us are cooking. So let me use a cooking analogy to describe the statistician's choice. You picked out a complicated French recipe for orange duck. When you start preparing the ingredients, you notice the duck breast has gone bad.

The statistician's choice would go like this:

(a) ignore the stench of the duck gone bad, follow the recipe, and pray that no one notices the stink

(b) run to the store next door, and, knowing they don't sell duck, buy their nice fresh chicken, substitute the meat in the recipe, and hope that the change doesn't ruin it

I always choose (b). But there are those who choose to hold their noses and do (a). Just like in the kitchen.

 

 

Excess deaths: why we want the cake and why we can't eat it yet

In my early March preview of statistical issues relating to coronavirus, I mentioned "people are counted as infected only if they are tested", and I also asked "how are deaths confirmed?"

For a while, the media ignored how meaningless the case counts were. Eventually, they got it, and that was great. But some experts convinced them that the savior is the death count. And now, in the last week, the reporters finally learned that death counts are subject to similar problems of definition.

The Economist just launched a tracker to look at excess all-cause mortality, using data from EuroMOMO. If during this pandemic, the total number of deaths from any cause is higher than normal, then it provides evidence (though not irrefutable) that Covid-19 is not a new name for influenza. Of course, we are finding excess deaths everywhere.

The next puzzle is that the number of deaths attributed to Covid-19 on death certificates is frequently lower than the excess deaths from all causes. This raises the possibility that Covid-19 deaths are under-reported. By itself, this is not necessarily true, because it is also possible that deaths from other causes are being under-counted.

However, it has also emerged that in many countries and regions, the published counts of fatalities so far have only included deaths occurring at hospitals for whom the deceased had returned a positive test for the virus. A good number of deaths occurred in care homes, hospices, and even private residences, and these have been ignored. In fact, many hospitals refused to admit patients who do not report severe symptoms, and told them to isolate at home.

The excess all-cause mortality data take a bit more time to collect, and perhaps not by accident, as these statistics come to light, many countries announced that they are revising how they count Covid-19 deaths, essentially admitting they were under-reporting.

***

The lesson related to data integrity is being taken up very slowly. Even as the Economist published the valuable tracker of all-cause mortality, they wrote:

... people have become grimly familiar with the death tolls... these numbers give a better indicator of a country's trajectory than do counts of confirmed cases, which largely measure how many people have been tested. Nonetheless, official covid-19 death tolls still under-count...

This is better than most but still repeats several fallacies:

a) Given the unknown fatality rate of Covid-19, the death count is a proportion of the case count. If data integrity is not at issue, the death count is a worse indicator of a trajectory than the case count because it just gives the same information after a time delay. If death tolls are better than confirmed cases, it's because we believe the data on deaths are less bad than the data on cases.

b) The emerging evidence shows that the unadjusted death counts are wildly inaccurate, so I'm not sure you can say the data on deaths are "the lesser evil".  In Lombardy, the Economist estimated the "true toll was about 120% higher" than the confirmed deaths. In Spain, the true number might be 60% higher (off a much larger base).

c) Confirmed cases is not a simple function of how many people got tested. More importantly, it is measuring who got tested. I know I have been hounding this point to death. In statistics, we care not only about the size of the sample but the composition of the sample! We could test lot of people with no symptoms and still have few confirmed cases.

d) The under-counting of deaths is still partially due to lack of testing. It is true that Covid-19 testing is not required in those all-cause mortality statistics but without testing, the count of deaths is clearly less reliable, just like the count of cases.

e) Deaths can also be over-counted, especially without testing to confirm coronavirus. How well can a doctor tell if the patient who died of pnuemonia had Covid-19 or some other illness?

f) If it's broke, you can't fix it. Countries change the definition of deaths mid-way but won't (and can't) change the past. This renders the death count series useless, unless you're willing to adjust the data. This was a key point I made about new metrics of obesity in Chapter 2 of Numbersense (link).

***

So, if you've been reading this blog, you know interpretation of statistics is no simple business.

While compiling these excess mortality numbers is necessary and important, here is something you should know, and I'm not going to sugarcoat it.

Imagine wearing glasses or sunglasses in the summer. You walk into an air-conditioned shop. You must wait for the fog to clear.

These excess mortality numbers are shrouded in a fog right now. It is actually somewhat meaningless to discuss them. This is because of a substitution effect. Some of the excess deaths we're seeing are accelerated deaths. (Of course, all excess deaths are accelerated deaths.) What proportion of the excess deaths are relevant will be up for debate once we wait long enough to see clearly.

A store usually runs an annual sale on New Year's Day only. This year, the store starts the discounts on the day before. After New Year's Eve, the store reports 50% jump in sales versus last year ("excess"). Was the promotion particularly effective this year? You don't know until after New Year's Day. When you compare the total sales of both days against the two days last year, you'll find that the jump is less than 50%. Some customers just showed up one day earlier.

 

 

 

 

Another note on fitting exponential models

This post is intended to provoke. It also contains further thoughts on the previous note I posted about the same topic.

(For less technical readers, I hope you persist and read the post. I try to make this stuff comprehensible for all.)

As the pandemic rages on, commentators talk incessantly about "exponential" growth. This growth model might be true in a theoretical sense but it is not reflected in the reported data. (I acknowledge up front that when the data do not fit a model, it does not follow that the model is incorrect for sure. That's a different post.) In this post, I make a couple of inconvenient observations around the exponential assumption.

It is easy to see that the real-world data do not show exponential growth in most countries. Here is the chart in log scale made famous by the Financial Times graphics team led by John Burn-Murdoch.

The interpretation of this chart as exponential growth is explained also here, at the Our World in Data site. They say this: "If, during an outbreak, the time it takes for deaths to double remains constant, then the disease is spreading exponentially."

***

Since we are talking about a model, let's make a model. Here are three exponential growth curves with three rates (A = 0.10, B = 0.15, C= 0.20).

When plotted in log scale, we see three straight lines. The slope of these lines are the respective rates.

This is one of the reasons scientists like to plot exponentially growing data on a log scale. The slope of each line gives the rate of each exponential. 

The rates are often expressed as "doubling times" in epidemiology. The following chart shows for each rate, the time needed to reach a given number of cases. The cases double from tick to tick. For example, for model A, it takes 3 days to double, and 6 days to quadruple. Getting straight lines means that the number of days to double remains constant regardless of starting point.

The following chart shows incremental doubling times. It takes 3 days to double to 4 cases, then 3 more days to double to 8 cases, and so on. It is a flat line for an exponential model.

This is the point made in the quote above.

***

Next, let's look at some real data. I'm using the Lombardia series since I have it handy from previous work with it. As I already showed before, in log scale, the growth in cases is clearly not exponential. It is not a straight line.

Many commentators adopt an interpretation that I will call "pointwise exponential". The curve above is interpreted as an exponential with changing rates (or, doubling times). In other words, at each point in time, it is an exponential with a specific rate. At a different point in time, it is a different exponential with a different rate. The rate keeps decreasing over time, which means the doubling time keeps increasing.

I am now going to show that this interpretation is misleading at all points in time. For illustration, I focus on 14 March, the date when the cases in Lombardia first exceeded 10,000. Computing the rate on 14 March is the same as computing the slope of the straight line that meets the curve on 14 March (for those who know, it's the tangent to the curve on 14 March). The slope for Lombardia on 14 March is 0.0755.

The slope gives me the rate, which gives me the doubling time. The rate of 0.0755 converts to a doubling time of 4 days. This means that I should expect the number of cases to have doubled from 14 March to 18 March. But the actual counts moved from 11,685 to 17,713, which is a 52% increase, not 100%.

The following diagram explains what went wrong. After I fixed the rate on 14 March, I'm projecting a straight line going forward (on the assumption that the growth is exponential). Since the actual cases dip below the straight line (shown as the dashed line), I over-estimated the number of cases on 18 March.

If the underlying data really followed an exponential model, then the actual case counts would skip along the dashed line, and this issue goes away. Since the underlying data are not really growing exponentially, the line bends away from the straight, and the further out I go, the bigger the gap.

The pointwise exponential interpretation basically fits a new line at each point in time. This means the slope is constantly changing. That makes the "doubling time" meaningless because by the time you get to the other side, the slope has changed.

***

Let me further provoke you with this next example. The following is another growth curve shown in log scale.

Instead of going to the extreme of a point-wise exponential interpretation, I use a piece-wise exponential interpretation. I treat each 7-day (weekly) period as an exponential, so that each week is fitted to a straight line, and the rate/slope/doubling time is allowed to change once a week.

Under this interpretation, this hypothetical region experienced exponential growth with weekly decreasing rates. It started at 0.11, then went to 0.04, and finally reached 0.02. Correspondingly, the doubling time started at 3 days, then went to 8, and finally reached 13.

This sounds reasonable (ignoring the problem I already pointed out in the first section).

Here's the problem. I played a trick here. The curve I showed you in log scale is actually a linear function, a perfect straight line. Here it is in linear scale:

Let me bring back the data plotted in log scale:

Should this growth curve be modeled as a straight line with a constant rate of growth or a piecewise exponential growth curve with decreasing rates?