One of the graphical obsessions of the Covid-19 era is the daily tracking chart, showing the counts of total or new cases , deaths, hospitalizations and so on. Later, people started talking about rates, and some charts show daily rates, e.g. of case fatality, or positive tests. For example, this chart in OurWorldinData:
The recent rising tide of Covid-19 cases in the U.S. conceals the bedrock of bad news: based on daily tracking charts like this, I have heard various people claim that while infections have grown, the death rate has been dropping. This trend is a statistical artifact because deaths lag infections.
On these daily tracking charts, the case fatality rate for any given day is the total number of reported deaths divide by the total number of reported cases. Among those reported cases are new cases, people who just tested positive. It usually takes one week to a month before a patient either recovers or dies. Any new cases, say reported within the last week, are extremely unlikely to contribute to death today but they will affect the fatality count in the coming weeks. So when an epidemic is spinning out of control, and infections surge, the proportion of reported cases that are less than a week old grows rapidly - these patients don't affect the number of reported deaths (yet), and so the case fatality rate as defined drops.
The solution to this problem is cohort adjustment. If its takes up to a month to know if a patient may die, then we track all the new infections today for the next month and at the end of the month, compute what proportion have died. In this definition, every infected person has equal amount of observation time.
One consequence of cohort adjustment is you must have patience. You reject the daily grind, the temptation to make bad "real time" decisions.
***
If you have run an A/B test, you most likely have fallen prey to the same fallacy. If you run a Facebook or Google ad, you typically set a time window, say a week. This means that the audience may see your ad on Monday, the first day of the campaign, or on Sunday. Most testing platforms show you a daily tracking chart. For any given day, they show you a "sales rate". That's the total number of sales from the beginning of the campaign up to that day divided by the total number of ad impressions.
A viral tweet sends a lot of traffic to the site suddenly driving up ad impressions but interest quickly dies down. Such traffic often takes a few days to a week to convert into sales. So the daily tracking chart shows traffic surging on the day of the viral tweet but the sales rate plunging! This doesn't mean these prospects are less likely to buy. It just means you need to wait for at least a week to give them enough time to make a decision.
When audiences see the ad on different days but they are measured on the same ending date, the daily sales rate is affected by the "age" distribution. If you apply a cohort adjustment, then the ending date will depend on the day of the ad impression, and vary by person. If your software does not let you do this, it's time to shop around!
Comments