On "Repeatability in Computer Systems Research"

Kudos to University of Arizona's Christian Collberg and Todd A. Proebsting for their article on Repeatability in computer systems research, appearing in this month's Communications of the ACM! It is wonderful to see a prominent call for our field to "fund repeatability engineering and reward commitments to sharing research artifacts."

This article recounts an interaction with authors of a research publication during an attempt to reproduce and build upon that published work; it is an interaction that is far too familiar! The article also provides a nice discussion of the benefits of reproducible research, repeatability, and benefaction. This sentiment inspired the authors' quest to review 601 recent publications for source code availability. The article provides a very nice visual summary of the authors' findings as well as a good overview of related studies to date and is well worth a read.

As one might suspect, the article reports depressing discoveries about the level of reproducibiity in scientific research. In response to these results, the article makes a somewhat controversial recommendation that is a great starting point for further discussion on reproducible research: to require authors to label their publications with "sharing contracts" that specify the level of repeatability of the work.

I find this recommendation an interesting point for discussion for several reasons:

*) The contract commits peer reviewers to take the promised resources into account when evaluating the contributions made by the paper. How will they be taken into account? We have already seen evidence that separate artifact evaluation committees are a good plan, rather than asking paper reviewers to do double duty.

*) What is the incentive for authors to include a sharing contract, or to include one that states they will share artifacts rather than not sharing? The article includes many strong arguments authors have expressed for not sharing in the past, so how does this contract address those problems?

*) Where would the implementation of sharing contracts start? At a particular conference like Artifact Evaluation Committees? As a journal publication standard? Elsewhere?


The authors also make a recommendation that I wholeheartedly support and would like to publicly endorse on this blog: "Funding agencies should encourage researchers to request additional funds for 'repeatability engineering,' including hiring programming staff to document and maintain code, do release management, and assist other research groups wanting to repeat published experiments." Well-said Christian and Todd! Please count me in on your efforts to make this happen!