If you have been paying attention to the scientific tabloids (if there is such a thing), the headlines would continually be filled with claims that modern science is having a crisis of confidence. In big bold letters, tabloids would announce there is a REPRODUCIBILITY CRISIS in science. But is all this due to a handful of grumpy scientists who got their last few papers rejected, or do the majority of scientists agree with the hype?

Online survey by Nature asks scientists about the reproducibility crisis

Over 1,500 researchers completed an online survey. As shown above, over half of the respondents thought there is a reproducibility crisis in science at present, while 38% thought the crisis is slight.

 

The survey showed many interesting, and at times surprising, results. For example, 70% of researchers indicated they failed to reproduce research results from other scientists, and 50% also failed to reproduce one of their own research findings! These responses, and the ability of researchers to publish successful and unsuccessful replication results are summarized in the figure below.

 

The survey also tried to identify what researchers believe is the cause of this reproducibility crisis (see figure below). The two most common responses were the pressure to publish and selective reporting of results (i.e., cherry picking). Other factors thought to be important were insufficient replication in the lab, poor oversight and low statistical power.

While identifying the problem is a start, another important aspect is what can be done to fix it. The survey found that many researchers and laboratories already incorporated strategies to increase the reproducibility of their research. However, the additional time and money required to incorporate these strategies were barriers to improving reproducibility. In addition, nearly 90% of respondents indicated a need for more robust experimental designs, better statistics and better mentorship. Similarly, 80% of respondents thought that funding agencies and publishers should do more to improve reproducibility.

The self correcting nature of science

With the pressure to published being so high, and the benefits of replicating previously published results so low, it is hard to see how the current practices that have led to this reproducibility crisis can be stopped. Although many practices to improve reproducibility are simple and, once in place, require little time to implement, it may be difficult to change the attitudes and practices of some senior scientists. Over the years they have learned the game of science and what is required to get papers published and grants funded. To change the rules of the game will likely be met with resistance, both by senior scientists, and also junior scientists who have learned the rules of the game from their mentors.

In the end, researchers need to decide whether they are doing science for the benefit of humankind, or the benefit of their CV.

Reference

Baker, M (2016). 1,500 scientists lift the lid on reproducibility. Nature. 2016 May 25;533(7604):452-4.