Big Data, Plainly Spoken (aka Numbers Rule Your World)

Is the U.S. speedboating? The merging of business and politics

In response to the note about the time lag between infection and deaths, Mark P. pointed me to an old blog post of his that features another example from the business world.

Mark's example comes from money lending. In that world, a key metric is the "charge-off rate". This is defined as the amount of new charge-off (i.e. bad loans) divided by the total amount of outstanding loans. Both numbers are computed as of the current date. What's the problem? Loans don't typically go bad right away. In fact, as Mark outlined, the industry won't even write off loans until it is already, say, 90 days past due.

And yet, investors evaluate lending businesses based on this "charge-off rate". How is this being gamed? Mark explains:

Let's say I book an account that goes delinquent after one year. That was a bad deal for the bank – – it lost money due to that decision – – but for one year, having that account actually lowered the bank's charge-off rate. Eventually, of course, this will catch up with the bank, but the reckoning can be delayed if the bank continues to book these bad accounts at an increasing rate.

I'd add that (a) the decision-maker has earned all his/her bonuses, and may not even be around for the reckoning, (b) the reckoning can be very far off so long as the bank has a pipeline of these risky creditors to lend money to, and (c) in the brave new world of massive corporate bailouts, the reckoning might show up as free money, more bonuses, and a stock price spike.

I'd further add that this practice - apparently having a nickname of "speed boating" - (a) is legal and (b) violates no accounting rules.

Of course, the whole mess can be avoided if the investors focus on cohort-adjusted charge-off rates. Computing the adjustment does require better data management.

***

Business executives are extremely good at crunching numbers. They may not know statistical theories, but they figure out what metrics investors are concerned about, and how to dress those numbers up. The U.S. President is a business exeuctive. He wasn't joking and he is not wrong when he said Covid-19 cases go up with more testing. (Don't forget deaths go up with more testing, too. And it's not how many you test but who you test that is the key to understanding those numbers.)

Let's talk about data backlogs

Yesterday, I featured some weird things I saw in Texas's Covid-19 data. A host of issues are officially recognized and flagged by sites like Covid Tracking Project. Reasons for bad data include software upgrades and data backlogs. News also came out that California "found" hundreds of thousands of unreported cases (link). This was blamed on a server outage necessitating code revision that stayed longer than it's welcome resulting in a data backlog. (I'm not joking; this came straight from the official explanation.)

If you haven't already, find a friend who has worked with databases, and ask if s/he believes these stories. Or, just keep reading...

 

Data Backlogs

If your job is to maintain a data pipeline that merges data from many sources, you will at a minimum create an audit report that tells you how much new data have been processed from each data source every day. If that is too much effort, you will make a report that tells you whether each data source has sent their data (yes/no) at the agreed-upon time each day. There is no database administrator who haven't made such a report. If someone hasn't, rest assured that these audit reports appear in the inbox of the CEO, the day after the first data processing error shows up, which probably occurs during the first week on the job.

It's very common, especially in modern, loosely managed distributed databases, that some minute or hour of data went unprocessed. Those are sometimes hard to detect from daily or weekly statistics in the audit reports.

It is simply inconceivable that hundreds of thousands of missing records have escaped attention. I laughed uncontrollably reading this delicious sentence: "the state hadn't realized they weren't receiving data from one of the largest commercial labs for five days." Oh, and the White House press corps didn't realize the President hasn't tweeted for five days. Missing multiple days worth of data goes beyond incompetence.

California is a repeat offender. I previously wrote about another such situation that happened on April 22, when on a single day, the state published 10 times the usual amount of data - to add insult to injury, this batch of data had one-tenth the usual number of positives.

 

Software Upgrades

Texas announced a data blackout on August 2, and then released two days' worth of data on August 3. A scheduled software upgrade was the official reason. But a software upgrade does not cause two days of data to get mixed together.

Data typically come with many dates. Two key dates are the date of the event (e.g. date of death, or date of diagnostic testing), and the date of the report. The reporting date is generally different and later than the date of the event. The systems at hospitals and clinics - the source of testing data - apply a time stamp to mark when the data are reported to the public health agency. The agency adds a time stamp recording when it uploads the data to its databases.

A software upgrade at the Texas public health agency isn't going to erase dates collected at hospitals and clinics. There is no reason why they can't save the data files for August 2 separately from those for August 3. Most likely, the hospitals and clinics probably have those files saved just like that, with the date of reporting as part of the file names. There is no logical reason why the two days of data must be aggregated.

Why couldn't the Texas public health agency divert the August 2 data to a temporary hard drive? Why couldn't it ask hospitals and clinics to send two files? Why couldn't it parse the dates in the data files to separate out the two days? Any of these solutions would have just delayed the release of the data but not scrambled two days of data. This is not rocket science. This behavior goes beyond incompetence.

***

Since the states blame these problems on server outages and software upgrades, they should name the vendors of these technologies. Which database needed an upgrade and why? What server broke down? If the upgrade was "scheduled", why wasn't there a contingency plan for storing the data temporarily in a different place? What measures are taken by the new technologies or vendors to prevent these same problems from happening?

Bad covid data: feature or bug in Texas?

Let me show you how bad the state of Covid-19 data is in the U.S.

Recall my post from two weeks ago about Covid-19 deaths in Texas starting to catch up with the earlier rise in reported cases. When cases started to spike in Texas, talk turned to the declining death rate, which is misleading since we should have been looking at cohort-adjusted death rates from the start. The media conspired to spread this misinformation, ignoring epidemiologists - when given a choice between reporting a misleading statistic today and reporting a more valid number in two weeks, the media cannot help but quench the immediate thirst.

In putting together the prior analysis, I already noticed the sad state of the data. On that day (July 27), Texas just changed how it reports deaths to rely on cause of death on death certificates. This created a single-day death toll of 675, well above the daily average, which presumably corrected for under-counting from previous days. With this correction, the state claimed that it will henceforth provide "more accurate" death counts.

What the state official did not explain is that the change in reporting also ensures a reporting delay for all deaths of at least one week. Let's think through this in the context of rising case rates and "declining" death rates. I make the nonsensical assumption that if patients die from Covid-19, they die immediately, on the day they are reported as new cases. Under the new reporting procedure, these cases will be reported today but the deaths will be reported when death certificates have been issued and can be inspected, which is at least 7-10 days in Texas. Thus, this group of patients (who already died) inflates the number of cases today while suppressing today's death rate since their deaths are not yet reported. This time lag is due to reporting procedures, and has a similar effect as the time lag between infection and death.

The data source, Covid Tracking Project, rated Texas "A" for "data quality" so presumably the state of the data elsewhere is worse.

***

I intended to update the analysis for this post last week but my effort was met with more data mischief. On August 2 (Sunday), Texas did not report any data, blaming a "systems upgrade" and promised that "data for Sunday will be posted with Monday’s data update." So I waited another week before publishing this update.

Just a reminder, two weeks ago, I presented the following chart:

The thin gray line is the growth curve for reported cases in Texas, smoothed using a seven-day window, from mid May through late July. The thick gray line is this growth curve shifted forward in time by three weeks. The red line is the growth curve of deaths in Texas, smoothed using a seven-day window. Except for the single-day anomaly (July 27), it appears that deaths roughly lagged cases by three weeks.

The next chart extends the time-line to August 1, the day before the data blackout:

This chart is disturbing. The growth of deaths in Texas has started to markedly outpace the growth of cases (time-shifted). Using the same time lag of 3 weeks, we should have expected deaths to climb 7 times between mid May to August 1. Instead, deaths from Covid-19 in Texas jumped over 9 times during this period.

This observation opens up a large can of worms. Back in July, the optimistic doctors and people were attributing the "declining" death rate not just to time delay but also to better treatments, younger people getting infected, virus becoming less lethal, etc. The data now suggest that treatments became less effective, or the virus got more lethal, etc. I'm not making such an argument, just pointing out why it is risky to draw premature conclusions from little bits of statistics.

***

Apparently, someone called time-out on these concerning numbers. On August 2, the state took a day off. When the data started flowing again, the death counts fell back to trend, as seen below:

For any data analyst, the alarm bells are deafening.

There are a number of suspicious issues here. Firstly, the official reason for the August 2 black-out was a system upgrade, and that the release for the next day would contain two days' worth of data. It doesn't appear that Monday's data contained two days' worth. The number of new cases released on August 3 was ~11,500. That is supposed to include cases for both August 2 and 3. Between July 27 and August 5, excluding Aug 2-3, daily cases were above 8,000.

Secondly, it should be simple to report deaths daily now that Texas is using death certificates to confirm deaths. I don't see how a system upgrade destroys the ability to know the dates of death. I wonder if they backtracked to the old way of reporting deaths after realizing the new procedure reveals a higher death toll.

***

Here's the problem facing data analysts in the Covid-19 era. The data are what the officials want them to be. They are constantly changing definitions, accounting rules, timing of releases, etc. It is impossible to make sense of the mess unless you maintain the database. Meanwhile, this data catastrophe is severely hampering public health decisions. The question is whether it is a feature or a bug.

 

 

Tautology in data arguments

One of the less discussed fallacies in statistical arguments is tautology. By this, I mean saying the same thing twice. Or, a conclusion that is true by assumption.

A recent example is the following popular argument:

(A) Our economy has suffered greatly because of anti-pandemic measures such as lockdowns, as evidenced by layoffs and GDP decline.

(B) Meanwhile, the deaths due to Covid-19 are insignificant - relative to the lost jobs. (*)

(C) Therefore, the anti-pandemic measures were useless and misguided.

The conclusion is that public health measures were useless.

Every statistical data argument has assumptions. What is a key assumption behind the jump from (B) to (C)? It is that the deaths due to Covid-19 as reported would be the same without the anti-pandemic measures that were imposed. Said differently, the assumption is that anti-pandemic measures are useless.

If one assumes anti-pandemic measures are useless, one will conclude that anti-pandemic measures are useless. So the argument amounts to saying the same thing twice. It is tautological.

Imagine if the opposite assumption were made, that anti-pandemic measures are effective at suppressing the death toll. Then, one will conclude that those measures are useful.

It turns out it's very easy to fall into the trap of tautologies in making statistical (data) arguments. That's because not everything is measured or measurable. Conclusions are an amalgam of data and assumptions. It's not always clear even to the analyst what assumptions have been made.

(*) Some article I came across this morning said there were 330 layoffs for every Covid-19 death in the U.S. This layoff-to-Covid-death metric is apples-to-oranges. All layoffs are counted as if there would not have been a single layoff in the absence of the pandemic (untrue) while only deaths that are confirmed with a positive SAR-CoV-2 diagnosis are counted in the denominator (under-counted).

***

Data science is not immune to tautologies. Here's one:

(A) We run an e-commerce website, and have developed an algorithm to recommend products to our customers.

(B) After the recommendation engine was launched, browsing (page views) of the top recommended products increased significantly relative to past trends.

(C) Therefore, our recommendation engine made the right recommendations.

The conclusion is that the engine recommended the right set of products. What is a key assumption behind the jump from (B) to (C)? It is that page views of other products would not increase similarly if they were recommended to the customers. Said differently, the assumption is that the engine recommended the right products to these customers.

If one assumes the recommendation engine works, one will conclude that the recommendation engine works. The argument amounts to saying the same thing twice. It is tautological.

Alternatively, one can assume that the engine is useless. Whatever is recommended to customers will get top views. In that case, one concludes that the recommendation engine is useless.

***

Have you come across examples of tautologies? Comment below!