Concerns about research reproducibility (or lack thereof) continue to escalate but it seems people can have quite different ideas about what research reproducibility means. In a perspectives editorial, Goodman and colleagues demonstrate concepts on reproducibility can be both precise:

…reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results … Reproducibility is a minimum necessary condition for a finding to be believable and
informative.

— U.S. National Science Foundation subcommittee on replication in science

or broad:

… a complex array of other factors seems to have contributed to the lack of reproducibility. Factors include poor training of researchers in experimental design, increased emphasis on making provocative statements rather than presenting technical details, and publications that do not report basic elements of ex-perimental design. Some irreproducible reports are probably the result of coincidental findings that happen to reach statistical significance, coupled with publication bias. Another pitfall is over-interpretation of creative ‘hypothesis-generating’ experiments, which are designed to uncover new avenues of inquiry rather than to provide definitive proof for any single question. Still, there remains a troubling frequency of published reports that claim a significant result, but fail to be reproducible.

— Francis Collins, Director, U.S. National Institutes of Health (NIH)

The authors suggest a framework to clarify different interpretations of reproducibility based on how reproducibility is described. Specifically, we can talk about reproducibility in three areas:

• Methods reproducibility
• Results reproducibility
• Inferential reproducibility

Methods reproducibility means sufficient details on study procedures and data are provided so that the same procedures could (in theory or practice) be exactly repeated. Results reproducibility is the ability to obtain the same results from the conduct of an independent study where procedures are as matched to the original experiment as closely as possible. These two types of reproducibility are sensible in principle but can be difficult to determine in practice. For example, the way in which cells are tested, on which machine, on what day, and how the machine was calibrated, can produce effects on some groups of cells but not others, but this level of methodological detail is typically not provided in publications and is not always requested by an investigator.

Inferential reproducibility means scientists can draw qualitatively similar conclusions from an independent replication of a study or a reanalysis of the original results. This is not the same as methods or results reproducibility because scientists might draw the same conclusions from different studies and data, or draw different conclusions from the original data, even if they agree on the findings (as evidenced by the ensuing debate after the 100 psychology studies were replicated). In a way, inferential reproducibility is much harder to determine but perhaps more important. The authors discuss how differences in statistical interpretation and incomplete reporting contribute to a lack of inferential reproducibilty.

In summary, breaking down the general concept of reproducibility into separate areas provides a simpler way interpret reproducibility and helps scientists identify and target areas to improve in their own research.

Reference

Goodman SN, Fanelli D, Ioannidis JPA (2016) What does research reproducibility mean? Science Translational Medicine 8:341ps12.