Scientific inquiry is the process by which new information is generated through experimental, theoretical or observational methods in order to understand the world. As inquiry progresses, some claims eventually achieve enough acceptance by the scientific community and become regarded as “fact”. The more established a fact is, the less likely it is subjected to further verification.

However, poor reproducibility of scientific findings and the potential for statistical misinterpretation have led some to question the veracity of scientific facts. Many studies are usually required to substantiate a claim, however the tendency for positive findings to be published in preference over negative findings means that published findings become biased by the selection of which articles are written and accepted for publication. If scientists are not aware of the degree of publication bias, they will fail to correct for this bias when making inferences from published findings.

In a recent paper, Nissen and colleagues mathematically modelled how misleading experimental findings and the presence of publication bias shape the creation of scientific facts. They demonstrate that publication bias (from the failure to publish negative findings) causes many false claims to become canonized as fact (Figure 1).

---

Figure 1: Receiver operating characteristic (ROC) curves where the probability that a true claim is correctly canonized as fact (vertical axis) is plotted against the probability that a false claim is incorrectly canonized as fact (horizontal axis), where the threshold to reject or canonize a claim is low (Panel A) or high (Panel B). Colour gradient indicates the rate at which negative findings are published. The graphs show two important features: (1) for any negative publication rate less than or equal to 1, the majority of true claims are canonized as fact, (2) when negative publication rates are small (less than 0.3 or 0.2 depending on threshold standards), many false claims will also be canonized as true.

 

This model highlights the following key points:

1. Publishing negative results is essential
2. Having a higher threshold to reject or canonize a claim does not reduce the need to publish negative results
3. P-hacking (testing or analysing data until non-significant results become statistically significant) dramatically increases the probability of canonizing false claims
4. Publishing negative findings even as a claim approaches canonization still greatly increases the ability to sort true claims from false ones.

The lack of publication of negative findings is likely due to a combination of investigators not reporting negative findings (the “file drawer” problem), and journals rejecting negative findings. Nissen and colleagues suggest this problem could be fixed by increasing incentivization by journals to publish negative findings (as some have done) or bypassing journals altogether and publishing negative findings on preprint archives.

Summary

When publication bias is present, false claims tend to be canonized as fact. This problem is reversible by ensuring that all findings from rigorously-conducted research are published.

Reference

Nissen SB, Magidson T, Gross K, Bergstrom CT (2016) Publication bias and the canonization of false facts. arXiv:1609.00494 [physics.soc-ph]