Researchers love p-values, especially when they are significant. It has previously been demonstrated that there is a disproportionate number of positive or significant p-values. In other words, many reported p-values must be false. That is to say, they are associated with false-positive findings.

In a recent article, Cristae & Ioannidis (2018) investigated the characteristics of p-values reported in figures and tables of three high-profile, multi-disciplinary journals: Nature, Science and PNAS. Results from 1997 were compared to those from 2017.

What they found was alarming, to say the least.

Audit results

Despite a similar number of figures and tables in papers published in 1997 and 2017, Cristae & Ioannidis found a marked increase in the number of p-values they included: 2.5 times for Nature, 5 times more for Science, 14.5 times more for PNAS.

Unsurprisingly, 94% of reported p-values were statistically significant.

On the bright side (or at least less gloomy), there was an increase in the number of papers that used corrections when performing multiple statistical tests. While such procedures were reported in only 2 audited papers from 1997, it was reported in 36 papers in 2017.

Why so many P-values?

In their paper, Cristae & Ioannidis (2018) highlight various factors that might have contributed to the increase in reporting of p-values in figures and tables, as well as the artificially high number of significant p-values that are reported.

1. Highly competitive, publish or perish academia
2. Highly incentivised delivery of positive results
3. suppression of negative results
4. p-values being the norm for reporting analyses results
5. More powerful statistical software providing more statistical analyses
6. More flexibility in the statistical tests that can be performed
7. More variables being analysed
8. More statistical tests being performed

Summary

The problem with p-values continues, and the current paper by Cristae & Ioannidis (2018) highlights that, at least when it comes to figures and tables in top-tier journals, the problem is actually getting worse.

Reference

Cristea IA, Ioannidis JPA (2018). P values in display items are ubiquitous and almost invariably significant: A survey of top science journals. PLoS One. 13(5):e0197440.