EPA commissioner Scott Pruitt sinks to a new low in his dogged campaign against evidence-based decision-making. (See my previous posts in which I dissected his other tactics here and here.) He directs his agency to propose new rules for disclosure of data and methods for climate science, placing himself in charge of opining whether a given study has enough disclosure to be allowed into evidence.
Slate’s Daniel Engber, someone I respect, weighs into this debate with a long-winded, confusing piece that loses its way. Pruitt tries to frame his anti-science crusade around the cloak of “open science/reproducibility”, an otherwise seminal movement that is rattling certain research fields such as social psychology. I encourage you to read what Engber has to say.
Here are a few things to bear in mind when thinking about this issue.
The type of data analyses that underpins climate science is of a different nature than the studies being attacked in the social psychology literature:
- The replication crisis concerns the failure to reproduce results from randomized, controlled experiments, intended to prove a cause-effect relationship. Data are collected after researchers design the experiments. A different set of researchers can follow the design to amass additional data in a replicated study. In climate studies, the scientists do not and cannot run experiments. Climate models use observational data, much of which are reconstructed or indirect measurements. These models are validated using different means, and so the proposed guidelines appealing to replication cannot be effective.
- The most controversial social-psychology studies make strong claims of small effects in the presence of high levels of uncertainty (for example, see this article about power pose research by Andrew Gelman and me). Andrew has a lot more to say about this research setting on his blog – in short, treat those studies with a huge grain of salt. By contrast, the key assertions made by climate scientists concern huge effects, e.g. the level of greenhouse gases in the atmosphere has not ever decreased since the start of the Industrial Revolution. More disclosure, or repeated requests for additional disclosure, will not alter that finding, but will delay the finding from reaching the public.
As I read Engber’s article, the words “manufactured crisis” kept popping in my head. There isn’t a crisis over climate scientists unwilling to explain their methods and reasoning. There isn’t a crisis over climate researchers sacrificing “quality” in the name of “quick” results. Quite the opposite: there is already more disclosure in climate research compared to other fields.
- The policy proposal has little to do with promoting “open science.” A huge number of scientists from diverse fields participate in the worldwide collaboration that produces the consensus climate models and results. If that is not “open,” I don’t know what is. The standard of disclosure is deplorable in many other fields: take, for example, all the grand claims about AI or machine learning coming out of industry labs, for which there has been selective disclosure on data, models, codes or methods.
- The policy proposal has little to do with increasing transparency in climate science. Year after year, thousands of scientists compile well-organized compendia of their models and results, making it easy to understand how they have come to their conclusions. By contrast, when there are commercial interests involved, data are often withheld or delayed – clinical trials data being a notorious example.
- The new rule moves the judge of what is “reasonable disclosure” from the community of scientists to the EPA Commissioner who has no expertise in research science. The status quo is obviously superior. Currently, a researcher who makes a bold claim must convince the community to accept his or her finding. If others ignore the study, due to inadequate disclosure, it will wither away. The proposal will make Scott Pruitt and/or non-experts decide what’s admissible, shoving the community of scientists to the curb. Engber argues that this problem can be solved by appointing a review board but he fails to convince me why a board of a dozen people would exceed the work of an entire community.
The EPA proposal is a set of remedies to manufactured crises that don’t exist. The reason behind this proposal can be understood in the lens of change management within organizations. Climate researchers are advocating changes in our approach to managing the environment using evidence from data analyses. The skeptics, including industries that fear suffering lower profits from adopting such changes, want the status quo.
This playing field is uneven, and it is played out not just in climate science but in every business decision in which the data analyses suggest that the status-quo strategies are deficient. To play the skeptic’s role, one doesn’t have to have any data, or theory, or anything at all. To keep the status quo, one just needs to stall by asking questions, requesting more information, ordering more research, etc.
It gets worse in the age of Big Data. By taking the data and conducting one’s own analyses, a skeptic can stall even better than ever before – it takes less time to generate useless analyses than to explain the flaws within such analyses.
If the EPA proposal is accepted, I’m afraid we are beginning to see the ill effects of “big data.” The nightmare scenario I outlined in my book, Numbersense, is taking shape. Here is what I predicted:
More data inevitably results in more time spent arguing, validating, reconciling, and replicating. All of these activities create doubt and confusion. There is a real danger that Big Data moves us backward, not forward. It threatens to take science back to the Dark Ages, as bad theories gain ground by gathering bad evidence and drowning out good theories. (p.13)
P.S. Andrew Gelman has a post about this and his readers react here.