Nature Communications has an inspiring piece about a DARPA project baking replication into the original study. The DARPA Biological Control program applies independent validation & verification (IV&V) to synthetic biology. The article is a Lessons Learned piece.
Although DARPA oversight presumably mitigated the element of competition, the IV&V teams had at least as much trouble as some historical cases discussed below. It’s worth reading their Hard lessons table. Here’s one:
And this one seems to apply to any discipline: “pick a person”
They also report how much they needed personal interaction — a result familiar to those of us in Science Studies (more later).
But I was still surprised how much.
To be fair, this is biology, a proverbially cantankerous field. But the canonical Science Studies reference of the difficulty of replication is laser physics.
Before I explore that, pause to appreciate the importance of this DARPA work: (1) The value of baked-in replication for really understanding the original result, and (2) the real difficulty in achieving it. I encourage you to read the NC piece and imagine how this could be practiced at funding levels below that of DARPA programs.
Echoes of the Past
In Science Studies, replication troubles evoke the “experimenters’ regress”. The canonical reference is Collins’s 1974 paper on laser physics (or really, his 1985 book):
Shortly thereafter, he notes that the many failures were:
This is of course echoed in the new NC piece about the DARPA program.
Collins expands on this and other episodes in his (1985), attempting to make sense of (then nascent) attempts to detect gravity waves. As Turner (2014) summarizes:
So the crisis isn’t merely that it’s hard to replicate from publications – any more than it’s a crisis that it’s so hard to learn to ride a bicycle by studying a manual. And no doubt many failed replications are failures of technique, caused by researchers foolishly attempting replication without contacting the original authors. The crisis is that we have many reasons to expect half the original results were indeed spurious. One of the goals of Replication Markets and the larger DARPA SCORE program is to help sort out which.
Back to the Future
I’ve fallen behind in the literature. I see that Collins has a 2016 chapter on the modern situation: “Reproducibility of Experiments: Experimenters’ Regress, Statistical Uncertainty Principle, and the Replication Imperative.” I look forward to reading it.
And of course this brings us to Nosek and Errington’s preprint, “What is replication?”, where they argue that replication itself is an “exciting, generative, vital contributor to research progress”.
But now, as Gru says, back to work.
Collins, Harry M. 1974. “The TEA Set: Tacit Knowledge and Scientific Networks”. Science Studies 4. 165-86. (Online here.)
Collins, Harry M. 1985. Changing order: replication and induction in scientific practice.
Nosek, Brian, and Errington, Tim. 2019-2020. What is replication? MetaArXiv Preprints.
Turner, Stephen P. 2014. Understanding the Tacit. (p.96)