Big Data, Plainly Spoken (aka Numbers Rule Your World)

The waiting game facing our decision-makers

This week saw a spike in COVID-19 cases in the U.S., and with it come the finger-pointing and blame game. First, fingers are pointed at other countries who made the wrong calls and let their cases spike, and possibly exporting the virus here. Then, blame falls on which state governor or mayor took the information seriously or not.

The reality is that each governer or mayor is faced with a very real challenge: the choice of action or waiting when there is very limited, and unreliable data in front of her/him.

I'm reminded of the following diagram, from Chapter 2 of my book Numbers Rule Your World (link). Half of this chapter looks at epidemiologists who monitor disease outbreaks around the country. They are the ones who sound the alarm when there is e-coli in our greens, which frequently lead to wholesale recalls of produce. Every week there are a few cases of disease but most of the time, they don't lead to large-scale outbreaks.

The left column shows what the analyst was looking at up to August or September in different years. In 2005 and 2006, she found a spike in cases. Each week, she had to decide whether to sound the alarm or wait another week. In 2006 (top row), if she didn't sound the alarm by late August, she would have been late to the outbreak, and the media would blame the CDC for underestimating the outbreak.

In 2005 (middle row), if she had sounded the alarm with the spike in September, she would have made a false alarm. The media would criticize the CDC for destroying the produce industry for six months or longer.

The point is, it is a tough call when you have limited data. It's pointless to slam an earlier decision after more data came in. Too much delay always looks bad if things get worse. Overly drastic action always looks bad if the worst did not materialize.

The most scary part of the situation - if you're the decision-maker - is the daily dread. Every day, new data arrive to judge your previous actions. As long as cases continue unabated, you'll be branded a failure. It's a situation that rewards careful interpretation of the data.

***

That's the lesson of Chapter 2 of Numbers Rule Your World (link), and it's here globally in an unprecedented scale.

 

Bittersweet

There is this bittersweet moment that all statisticians and data scientists experience in any successful project. It's the moment where we finally cleared the forest, and saw the light. It's the moment for claiming our prize -- a simple, coherent story explaining the data mess. Now that we've got the map of the forest, it's simple to retrace the path to the light. We wonder how we took all those wrong turns. The taste is bittersweet.

***

I thought about all that while hearing the know-it-all pundits shouting at public health officials, epidemiologists, and politicians for not acting fast enough, for ignoring the dangers, for letting the pandemic grow so rapidly.

One simply cannot look at the coronavirus growth curve today and complain about a decision made two weeks ago; that was two weeks ago when we didn't have the two weeks of data we do today. Everything always feels simple after you've seen the data you didn't have.

***

Consider two countries A and B. Country A gets its first case, and eventually the virus arrives at country B. When country A made decisions related to containment, it had only five days of data. The cost of containment is likely economic collapse. It chose to wait and see.

Then, the case counts started to jump, and country A took more drastic action after 10 days. In this period, Country B started seeing its first cases. It faced exactly the same dilemma as Country A. Case counts were low by day 5 and growing slowly. Drastic measures would destroy the economy. It chose a wait-and-see approach.... until the cases began to rise sharply.

Country B blamed country A for not moving fast enough to contain the virus, thus allowing it to spread. But on day 5, it was not certain, not even close, with the limited initial data, that the contagion would be become serious. The only reason why Country B felt wronged is because the virus got there later. Even with the benefit of more data, Country B mimicked Country A's decisions. In fact, one can say Country B's action validated Country A's earlier decision to stall.

***

When we finally emerge from this crisis, we will reach that bittersweet moment. We've conquered the virus, but we'd lament that we didn't do it sooner. That's the deceptive power of hindsight.

 

Note about fitting and visualizing exponential models

One of the consequences of the open data, big data movement is that everyone is an amateur data analyst. To wit, Andrew Gelman earlier linked to a Kaggle competition to forecast coronavirus cases.

This is both unnerving and exciting; it's unclear if the benefits of more eyes outweigh the costs of misinformation. But there isn't a way to push the genie back in the bottle.

I don't plan on spending much time with the currently available data about the coronavirus cases, mainly because there is too much missing context, such as the amount of testing, what types of people get tested, how cases and deaths are defined (as related to the novel coronavirus), what measures are taken that impact the counts, age and other co-variants, etc.

The one analysis I did from the other day addresses the narrow question of whether there are signs yet that the containment measures yielded results in the region of Lombardia. Zooming in to a single region is helpful as the variations due to definitions and policies is limited. The key takeaway from that analysis is that the growth curve of reported cases is not exponential.

This note is written for those attempting to fit exponential growth curves.

***

A typical process of fitting exponential curves is to plot the data with a log y-axis. The supposed benefit of looking at a log plot is that the implied growth rate can be eyeballed from this chart. The hockey-stick curve of case counts looks like a straight line in the log scale. The slope of this best-fit line leads to the growth rate of the exponential model. 

Here is what happens when I obtained a "trend line" in Excel after transforming the case counts to a log10 scale.

For this illustration, I used the entire data series (Feb 25 - Mar 19, 2020) to fit the model. Excel reports that this is a tight-fitting model, with R-squared of 98%. [See my previous post for why using the entire data series to fit a model, as I did above, isn't recommended.]

The measure of goodness of fit (R-squared) comes from aggregating individual daily errors. In the following chart, I highlighted two such errors, the gaps between the model estimate and the actual case count on two selected days. I picked those two days because the lengths of the two red lines are almost identical.

For a normal linear regression fit, it's standard practice to plot these errors and conclude that they are pretty uneventful.

The log scale makes huge errors look small.

It's extremely easy to forget that we are looking at a log plot here. Log transforming the data has the effect of pulling big numbers closer to zero, and pushing small numbers further from zero. If the two red lines were plotted in the original scale, you'll see that the error on March 19 is much, much larger than the error on Feb 26!

This misreading of the error sizes is the same visual misperception that makes any log scale prone to misinterpretation.

***

The above problem is not limited to the phase of model fitting. It is common today to draw two (or more) growth curves in log scale and compare them. This is a typical visualization:

This one from Spanish outlet El Pais (link) was featured by Alberto Cairo (link) recently so I have it handy - but my comment applies to all variants of this chart which can be found everywhere. The El Pais design is distinguished by a side-by-side presentation of the growth curve in log scale and in linear (everyday) scale.

The log scale complicates comparing two growth curves.

When comparing two growth curves, say Spain and Italy, comparing the straight-line slopes is fine (with the caveat that we are then assuming exponential models for both countries).

We get in trouble when we interpret differences in slope over time. Using the above graph, one might say that in the first ten days, the Spanish case counts were lagging behind the Italians but after that, there was a cross-over and the Spanish cases grew faster than the Italian cases.

All those words represent our interpretation of the gaps between two growth curves. The optical illusion described in section 1 of this post applies here equally. Gaps to the left side of the time-line are artificially magnified by the log transform while gaps to the right side are artificially compressed.

The further out it goes on the time axis, the bigger the compression. You have to train your head to think in log scale to reverse the visual distortion.

The reality is only experts who have been trained to think in log scale can properly interpret this type of chart.

 

 

How to act like a data scientist 8: Don't use lagging indicators to forecast

It's ever more important to apply your numbersense when you read the news about Covid-19. This past week, many in the media continue to obsess over horse-race metrics, i.e. current counts of cases and deaths. Those happen to be the key data being released and updated by governments around the world, but raw data rarely tell the full story, and in this case, they mislead.

I already wrote about the problem with missing data that have caused foolish decisions to move to "safe" places. (Is it safe or do they not test?) You can't compare the growth curves of U.S. with China, or South Korea, or even Italy with a straight face. The elephant in the room is the extent of testing. If you test less, you under-report infections. Not enough testing under-estimates the infection rate but exaggerates the mortality rate. Under-reporting the infection rate is a risky business - it makes people complacent.

This past week, the news headlines have been blasting: Italy breaks records in number of deaths. This is not fake news but it is misleading. The message sent to readers is that the situation is spiralling out of control in Italy. But more new deaths does not mean things are getting worse! New deaths is irrelevant to understanding whether the containment measures in Italy is working or not.

The number of deaths is a lagging indicator. We can predict the growth in deaths based on the reported growth in cases in the last week. If there was a surge in cases, there will follow a surge in deaths... after a lag. When I hear new deaths went up, I think new cases went up last week. It provides little new information.

***

Is there a better answer to whether containment measures are succeeding?

First, we can zoom in on Lombardy (Lombardia), the region where the initial outbreak hit. Look at the growth of cases over time. The raw data can be found on this site (the green line is Lombardy).

Growth Rate is Not Exponential

The first surprise I got from looking at this data is that the green line is not exponential. An exponential rate can be expressed as something like "cases increase 10-fold every X days". The actual data aren't quite that bad! In math terms, the rate of growth is polynomial. It's still greater than linear but not as bad as exponential. Linear growth can be described as "cases increase by 10 every X days". [Note: Ten and ten-fold are hugely different rates, leaving a huge space in between.]

The chart below shows the data from February 25 to March 19, 2020 fitted to three rates of growth (quadratic, cubic and exponential). The exponential curve started to bend away from the actual case counts, and the magnitude of the over-estimate will keep growing over time. It's just the wrong shape to use here. The cubic and quadratic curves (both polynomials) are both plausible, the former being a tad better.

It Fits the Past but does it Predict the Future?

Now, take a step back to March 8, when the government decided to lock down the entire Lombardy region; some counties have already been taking actions prior. Could it be that at that moment in time, the data showed an exponential rate of growth? Did the shape change after the lockdown?

In the next chart, I used the data up to and including March 8 to fit the three rates of growth (quadratic, cubic and exponential). Then, for the period March 9-19, the curves show what the case counts would be given the assumption of that rate of growth (to be exact, it's the shape of the growth curve). In other words, the lines after the lockdown are forecasts that can be compared to the black dots which represent the actual case counts.

You can see that the exponential curve quickly leaves the data (black dots) behind. The actual growth in the cases is far below what an exponential rate would have predicted. Here, you see that the cubic is actually the best fit.

[For the nerds here, the two quadratic curves are different. In the first chart, the quadratic curve is fitted to all the data. In the second chart, the quadratic curve is fitted to just the pre-lockdown data, and then extrapolated. This is an example to show why predictive power is important to measure model fit. Quadratic fits in both cases have great R-squareds but that doesn't mean it is the right model.]

So we have both good and bad news for the italiani. It says that the growth is not as frightening as the media breathlessly reported. But it also says that the curve hasn't flattened (which is why the Italians just announced they would send the Army to enforce the lockdown.)

***

A Glimpse of Hope?

However, an analysis like the above is self-fulfilling. The model does not allow for the shape of the growth curve to change over time. I just assumed one shape which means the curve could not flatten under this analysis!

To escape from this dead-end, I looked at just the period after lockdown. The following chart shows the case counts from March 9 to 19.

This is when you can say "Andra' tutto bene!" In the last 10 days, the best-fitting model is linear. (Given margin of error, the quadratic curve can still be possible. But certainly not the exponential!) In this sense, you can say that the measures taken by the government in Lombardia has flattened the curve. The shape of the growth has been dampened from more like cubic to more like linear. That is an achievement, although the battle hasn't been won yet.

[Similar to the above analysis, we would need to observe the trend for another week to know if this linear model has predictive power.]

***

Meanwhile, we have also some anecdotal reports from specific towns, showing that containment and testing strategies have worked. (Lodi, Bergamo, Vo)

Of course, other indicators should be analyzed to paint the full picture. Also, the other regions are lagging so the hope is that by taking steps earlier, their curves would look milder than Lombardia's.

The point is that new deaths breaking a record is not a useful indicator of anything (other than the fact that infections broke records the week before).

 

P.S. Credit to JMP software for making it so easy to analyze the visualize the data. All the data processing, model fitting, visualization in one place.