For science to move forward, it is essential that key research findings be reproducible. However, current publication and career progression models do not value replication. It is almost guaranteed that a replication study will be published in a journal with a lower impact factor than the journal where the original study was published.

As pointed out by McLoughlin & Drummond (2017) in a recent Experimental Physiology editorial, replication studies are often rejected on grounds that they lack novelty, because novelty is a key requirement for consideration of a manuscript by many journals. To help reverse this trend, Experimental Physiology is leading the way by introducing a new category of research article focused on replication.

Replication

In their editorial, McLoughlin & Drummond (2017) suggest that important new findings must be replicated by others before being accepted by the community, but discoveries of lesser importance may be of little value and thus not warrant replication.

Regardless of the importance of a discovery, replication studies have historically been difficult to publish in high-end journals. Thus, in a publish-or-perish research environment, there is little incentive for researchers to undertake a replication study.

A new category of research publication

Because of the current difficulties and lack of incentives to publish replication studies, Experimental Physiology has introduced a new category of research publication. Submissions that adhere to three criteria established by the journal will be published regardless of whether the findings agree with or contradict the original report.

As you will see, the three criteria ensure that replications studies make a useful contribution to their field of study.

Criteria 1: Target novel findings of physiological importance

Replication studies submitted under this new publication category must seek to reproduce a novel finding that is considered an important advancement in the field of integrative and translational physiology; a finding worth confirming.

The study should be undertaken by an independent research group and follow as closely as possible the methods used in the original publication.

Criteria 2: Make all individual subject data freely accessible

The criteria ensures that the data can be used in any future study that combines data sets to establish more robust estimates of effect sizes. This approach, which can help confirm the importance of a novel finding, is called individual participant data meta-analysis in other fields.

Criteria 3: Sufficiently large experimental groups

The proposed replication study must have sufficiently large experimental groups to make a meaningful contribution. One option proposed by the authors is to have an n=10 in each experimental group. This recommendation seems geared towards animal studies. The other option proposed by the authors seems more appropriate for studies involving human subjects: the number of subjects in each group should provide sufficient statistical power to detect the important effect reported in the original study.

In their editorial, McLoughlin & Drummond (2017) remind readers that a perfect replication study with the same sample size has only a 50% chance of discovering a significant difference (p<0.05) between groups.

In most cases, replication studies should have greater statistical power to detect an effect than the original study. These studies should also seek to detect an effect size that is less that the first reported.

Summary

While this editorial points out that this type of replication study will help physiological researchers to decide whether a novel finding is real and large enough to have general physiological importance, it will be interesting to see how many of these papers are published in the coming years. This new category of publication does provide a new opportunity for researchers to publish important replication studies. Ironically, it does mean that researchers conducting a replication study will likely have to do more work than the authors of the original study. In a system where novelty is valued above all else, what is the incentive for researchers to undertake such studies?

Reference

McLoughlin P, Drummond G (2017). Publishing replication studies to support excellence in physiological research. Exp Physiol 102: 1041-1043.