I gave this talk yesterday at the Digital Studies for Digital Science (DS2) online conference. You can watch the recording on YouTube.
Open science can’t solve the replication crisis
In the last few years, concerns about the replication crisis in biomedicine and social psychology have bolstered the open science movement, and played a significant role in arguing for open science requirements at scholarly journals and even government agencies — as in the case of the US Environmental Protection Agency’s “Strengthening Transparency in Regulatory Science” proposed rule. However, the discourse surrounding the replication crisis frequently conflates two very distinct desiderata of experimental-computational science, namely, replicability and reproducibility. Following definitions proposed by the US National Academies, reproducibility is a purely computational notation: the ability of an independent researcher to produce numerically identical output from an analysis, using the same data and analysis code. By contrast, replicability is the ability of an independent researcher to reach qualitatively similar conclusions by repeating an experiment, using the same analytical approach but collecting different data. I first argue that reproducibility has minimal (but non-zero) epistemic value, comparable to mere logical consistency. Next I survey a variety of proposed causes for the replication crisis: p-hacking, publication bias, insufficient statistical power, unrepresentative samples, publish-or-perish incentive structures, noisy measurement, underspecified phenomena, imprecise theory, data mismanagement, software bugs, and fraud. I argue that open science requirements effective promotely reproducibility, but promote replicability only insofar as replication failures are due to causes that leave traces (as in historical science) in data and code. Because very few of the proposed causes leave such traces, open science cannot solve the replication crisis.