99.4% art, 0.6% data

The media could not conceive how the CDC could revise its estimate of the proportion of omicron variant so drastically from a heart-stopping 73% to a blood-curdling 59% in a matter of two weeks (for example, Bloomberg scratch that since you can't even read one article on Bloomberg. Here's MSN, so 90s.)

The reason why the media is surprised, stunned, shocked, dismayed - is because the media didn't do its homework when they excitedly reported the 73% number.

I knew because I hopped on the CDC page that contained this number. From there, you immediately learned that 73% is a "Nowcast", which is described as "a model that estimates more recent proportions of circulating variants and enables timely public health action". In plain English, it is a forecast, not actual real-world data.

My first instinct when I see a model (because I build models for a living) is to click the very helpful button that toggles between "Nowcast on" and "Nowcast off". You can't understand any model without first looking at the actual real-world data sitting beneath it.

I was indeed surprised, stunned, shocked, dismayed. Because this was what I found (these screenshots were taken before the latest revision):

The orange section is the Delta variant. The tiny slither of purple at the bottom of the very last column is the Omicron variant. On the table, you see that the actual proportion of Omicron in the week ending Dec 4, 2021 was 0.7%.

The next screenshot was taken when Nowcast was turned on:

The last column showed 73% Omicron, which was all over the news when this came up. Notice that the date axis changed. There are two additional weeks shown: ending Dec 11 and Dec 18. The 73% apparently concerned the week ending Dec 18.

It appears that "Nowcast" is not really a forecast but a missing data imputation procedure because this information was released right after Dec 18. This CNet news article was dated Dec 20. Presumably, the flow of data did not support real-time reporting, and so they had to resort to a model.

***

What is this Nowcast model that can aggressively turn 0.7% to 73%? Unfortunately, your guess is as good as mine. The link behind the word Nowcast on the CDC page leads to the chart itself. There is nothing on the chart that explains what kind of model is Nowcast. I found nothing on the page that explains how they turned 0.7% to 73%.

But we can measure how horrible this Nowcast model has performed. The media got this wrong too. It's not 73% versus 59%. Look at the current view of the chart with Nowcast on:

The 59% estimate is for the week ending Dec 25 while the 73% estimate is for the week ending Dec 18. The correct comparison is 73% versus 23% (the purple section of the second column from the right). They "projected" a 10 100 fold increase but now they say it was a 3 30 fold increase. No wonder they didn't want to tell us what this model is!

***

To take a Bayesian perspective, the model estimate is a kind of weighted average between past data and "prior" knowledge. In this case, the prior knowledge is "art" reflecting someone's subjective belief. We don't know much about the model but we know that this prior belief exceeds a 10 100 fold increase because it cancelled out the past data (0.7% of cases) and more.

Science in the pandemic age is just like this. Scientists running away from other scientists who are capable of evaluating the science.

 

[12/29/2021: correcteed 10 to 100. 12/30/2021: added "the proportion of" in the first sentence, responding to Antonio's tweet. 1/2/2022: corrected 3 to 30]

 

 

Comments

[Michael Droy]

To be quite honest if I was tracking a number at 0.7% and forecast it would be 73% in two weeks - I'd be feeling hugely proud of myself if it turned out to be 23%. Doubly proud if it went on to be 59% on week 3.

0.7% to 7% to 23% to 53% is x10, x3.3, x2.3

0.7 to 73% is x104 or x10.2 per week.

So even accurate for week 1. (I know you can't treat %-ages as exponential models because they stop at 100%, but crudely and below say 40% it kinda works.)

The modellers should be congratulated, not mocked.
Whether the predictions should have been published the way they were is a different question.

The real target of mockery should be the idea that Omicron appears to be a threat when all the evidence is very very low hospitalisations and almost no deaths. England reported 49 deaths to date (yesterday) but we know most Covid hospitalisations are arriving for other causes so many of this 49 will be "with" and not "by".
Omicron seems to be a blessing.

[Kaiser]

MD: I don't know what to make of your comment. Maybe you are being sarcastic. But you just made me realize I gave them too much credit. From 0.7% to 73% is not 10 fold as I said, but 100 fold.

They are making the same mistake that they have been committing from day 1 of this pandemic. You cannot compare the # of deaths and the # of cases on a cross section of time. That is assuming zero lag between cases and deaths.

The U.S. has seen an average of 1,000 deaths per day for three consecutive months this far into the pandemic. That is a run rate of 360,000 deaths a year. They also told us that death rates of Delta were lower. Somehow this is not reflected in the aggregate numbers. Strange, no?

[A Palaz]

Does US test also use just the S gene dropout result to see if Omicron?
I remember this test pattern also used to show the UK variant last year.

But I also remembers that an S dropout can happen at higher CT.meaning only N and ORF or E are seen.

[Anonny]

Quibble: You corrected the 10 -> 100, but forgot the 3 -> 30 in the third last paragraph.

**

It really is terrible that they don't have any easily accessible documentation of the model - or even, dare I dream, the code. It may be a bad model, or it may be a case of small errors in the input leading to large errors in the output, like what we see happen sometimes with SEIR models.

I'm less concerned with it being bad; even "failures" can be informative, in that they can teach us what pitfalls to avoid in the future. It's hard to learn much of anything here. This would still apply even if the nowcast was accurate.

[Michael Droy]

Kaiser: No. I think you want to consider what a good forecast would have been like and whether proportionately it would have been any better.
They predicted a rapid exponential rise in two weeks and a rapid exponential rise happened in three weeks.
To suggest that a 100x forecast is inaccurate misses the point. No one could have predicted a 30x rise accurately. to may any prediction of >10x is both a bold call and a good one.

They are making the same mistake that they have been committing from day 1 of this pandemic. You cannot compare the # of deaths and the # of cases on a cross section of time. That is assuming zero lag between cases and deaths.
Maybe I missed something - I can't see any reference to deaths in the forecast.

[Kaiser]

MD: They did not predict a "rapid exponential rise" in some kind of handwaving, qualitative sense. They made a numeric prediction, which is embarrassingly wrong. And they chose to hide the model so no one knows how they came up with such a horrible prediction.

[Michael Droy]

What would a "good" prediction have been for something that went from 0.7 to 23?
Would 5 have been more embarrassingly wrong? I think yes, 5 would have been worse.