Data autonomous arguments are not data driven

I love "data-driven" arguments. You probably do too if you're reading this blog.

What does it mean to be "data-driven"? One prerequisite is that the conclusion should change as the data change. The analog that I write about frequently on the dataviz blog is to qualify as a data visualization (as opposed to just a visualization), the visual should change as the data change.

***

Recently, there have been numerous arguments that are immutable in the face of changing data.

John Burn-Murdoch at FT wrote this tweet: 

John dislikes this argument:

If there are spare beds, this proves ex-post that mitigation measures were not necessary.

There are different potential problems with this argument.

John is talking about limiting factors. In a complex system, many factors combine to produce an outcome. Typically, only one or a few factors are "limiting" in the sense that if those factors were either relaxed or tightened, the outcome would be directly affected. Meanwhile, there are many other factors for which manipulating them would not alter the outcome. These other factors may matter under other combinations but they don't matter right now. 

Another counter-argument is related to the direction of causality. One reason why the government may shut Nightingale hospitals (for those in the U.S., these are akin to the makeshift beds set up at conference centers) is because mitigation measures were successful in reducing the demand.

Yet another counter-argument is the assumed counterfactual. Because mitigation measures were taken, we could not observe what could have happened if they weren't. The absence of evidence is being taken as evidence of absence. 

***

But I think the best way to counter this type of argument is to point out its inherent data-agnostic autonomous structure:

I) On the one hand, if the outcome is X, then we conclude Y.

II) On the other hand, if the outcome is not X, then we conclude Y.

In other words, in all cases, whether X or not X, we conclude Y.

 

In the concern over Nightingale hospitals, this structure translates to:

I) On the one hand, if there are spare beds, then we conclude that mitigation measures were unnecessary, causing harm without benefit.

II) On the other hand, if they run out of beds, then we conclude that mitigation measures were ineffective.

In all cases, we conclude that mitigation measures were unwise.

The problem is that the same conclusion results regardless of the data (need for spare beds). So this is not a data-driven argument despite the presence of data.

 

***

The Bayesian way of thinking explicitly endorses the premise that conclusions should change as the data changes. Bayesian analysis leads to a posterior probability estimate, which explicitly depends on the data. If you feed the analysis different data, you get a different probability distribution. For an example of this, see this post in which I showed how the Pfizer vaccine efficacy estimate changes, assuming different trial outcomes.

The classicial (frequentist) method of hypothesis testing also produces different conclusions given different data inputs. The "null" hypothesis does not change but with different data, the location of the observed sample with respect to the "null" distribution changes, leading to different p-values.

 

[P.S. 1-1-2021. I switched from "agnostic" to "autonomous" as I think the argument being autonomous of the data conveys the meaning better.]

Comments

[Michael Droy]

I get the lack of data argument.
I agree that argument A can only be said to be supported by data if Not A would imply a different result.

But there is still a lot wrong with how the original claim has been denied.
Most recent count is that UK has 21,787 patients in Hospital beds with Covid.
There are 112,000 doctors and 312,000 qualified nurses in the NHS (out of about 1 million total staff).There is no shortage of Doctors or nurses for Covid. What exists is a determination to continue to treat other patients rather than re-assign them to Covid patients. And that is because Covid is a limited problem which takes up only a small fraction of NHS time. The original claim is valid.

Actually it is Covid lockdown practices that really use up NHS capabilities, as every face to face treatment now takes up to 3X what it used to, so vastly reduced services are available while something like 20% of the NHS is devoted to Covid patients. Last number I could find was 140k hospital beds in UK

[Kaiser]

MD: It's useful to think about what other paths lead in or out of your train of argument. For example, if there were indeed a crowding out of beds for other patients, then the vast expansion of bed capacity via these Nightingale hospitals should have dealt with that. You'd simply move the covid patients into these facilities and let the non-covid patients be treated in the usual wards. So, can I now argue that the removal of the extra beds is evidence that other patients have not been crowded out?

This illustrates a huge difference between computation and statistical inference. To establish cause and effect, we need to go beyond chaining up a series of numbers. A + B = C may work numerically but to establish cause, we need to know that A and B are the only factors that influence C, that A and/or B directly influences C without passing through another factor, that C doesn't change on its own, etc. etc.

[Michael Droy]

Well none of the numbers work, because at heart the Covid lockdown has slowed down the NHS much more than Covid patients have taken up capacity, so all measures are actually measuring lockdown policy, not virus effects.

I think my problem with the tweet is that the author is attacking a correct claim (The situation is not as bad as many claim) with a completely nonsensical counter claim. The first claim may be based on data that would also fit an alternate hypothesis. But the counter claim (there are not enough nurses) is actually based on an entirely different limiting factor - covid lockdown and restrictions on working practices.*

The truth of the matter is that while there are enough beds for Covid patients treatment for any other illnesses is almost impossible to obtain. Seeing a family doctor is almost impossible. Non-urgent operations are all cancelled. There is little early testing of cancer occurring.

*It is not the virus that is damaging the NHS, it is the over-reaction to the virus - which given how Covid kills patients is quite ironic but not funny.