Preparing computer code for peer-review: Nature journal guidelines

Computer code is used to analyse data in research studies across many fields including epidemiology, biomedical science, computational biology and physics. Many findings now depend on such analyses. What role do journals play in ensuring transparency and reproducibility of computer code used to generate research findings? How might this fit in with our efforts, as scientists, to reduce errors in science?

In early 2018, Nature published an editorial stating that Nature journals would require editors managing papers in which computer code is central to key findings, or is part of the key finding of the paper, to ask reviewers to peer-review the code on a case-by-case basis. In this way, the Nature family of journals encourages researchers to include custom-written programs for peer-review.

To help researchers, Nature developed a checklist for code and software submission. Here are the documents the journal requires for peer review of computer code:

  • Compiled standalone software and/or source code including version details and a README file listing all the documentation provided as detailed below. Please submit as a compiled zip file or provide a link where all the necessary materials can be accessed.
  • Installation guide that includes information on the operating system, programing language, software dependencies and non-standard hardware or resources needed to run the program and details of typical install time on a current computer.
  • Demo that runs the code/software in example data and typical run time.
  • Provide a link to the code in an open source repository and a digital object identifier (DOI); when available.
  • License of use; we recommend using a license approved by the open source initiative. Please note that an open license for code published in association with a Nature journal paper is compatible with the terms laid out in the Nature journal License to Publish.
  • We strongly recommend that you ask colleagues that are not familiar with the tool to test it prior to submission.

Wow! That seems like a big ask. Just as well most of us are not submitting a paper to Nature tomorrow. We may never have submitted computer code for review before, or considered doing so. Or we might be new to research with projects that will require learning to write computer code, and feel a little overwhelmed by it all. Let’s take a step back: what overall principles should we consider when preparing code for submission? How could we build in this preparation early on in the research project so we save time at the end?

In general, the aims are to have computer code run and analyse the data with a simple “button press”, and document the code clearly (but not overboard) so that someone outside of the study can follow the analysis process. This is what is meant by software being standalone.

Consequently, this means any computer packages that the code depends on should be accessible either directly on submission or through links, so reviewers don’t have to hunt them down. Also, we should include a document (a README text file; i.e. the first thing you read) that explains how the code is structured, how to run it, and what the expected input and output files are.

Even if you stopped here and managed to assemble these components for submitting computer code for peer review, you have achieved a lot. If we push on, other principles include making code transparent (by publishing it on an open source repository), adding a license to our code to make it clear who can it and how, and providing instructions and worked examples of exactly how the code is run (perhaps more relevant in heavy-duty computational science). To this end, our Scientifically Sound series on public data and code repositories, licencing data and code, and packaging computer code for research projects provide some guidance on how to implement these practices.

While Nature now implements these requirements, the journal also recognises there may be exceptions to the rule. For example, full disclosure is not always possible (e.g. in commercial applications where computer code is proprietary), supercomputers may be required or run time is lengthy, and computer code may not be feasible yet in some fields. In these cases, editors and reviewers will assess how best to present and publish computer code.

We hope that you can recall these principles to mind as you start or continue working on your code in your research projects.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s