Reporting bias (ie. bias arising when dissemination of research findings is influenced by the results) is thought to be common in biomedical and medical research. However, exactly how common it is has been difficult to quantify. Albarqouni and colleagues examined how commonly reporting bias occurs by examining the distribution of p values in medical research studies, and compared these distributions across clinical areas.

The investigators randomly selected 1500 PubMed records and extracted data from 758 of them: data included study type, study design, medical discipline and p values for the first reported outcome and primary outcome (if specified). They found that primary outcomes were only specified in 51% of included studies, and the first reported outcome differed from the primary outcome in 28% of included studies. Histograms of the distribution of p values showed an excess of published p values at common thresholds of 0.05 and 0.01 statistical significance. The excess of significant p values was common across medical disciplines, study designs and and study types (Figure 1).

---

Figure 1: Histograms of z scores and associated p values for first reported outcomes across medical disciplines. Dashed lines indicate common thresholds of p = 0.05 and p = 0.01 for significance tests; k, number of studies; N, sample size. (Figure 3 in paper.)

 

 

The presence of reporting bias in research has serious implications for the validity of systematic reviews because published findings are not representative of all conducted studies and analyses. The investigators suggest strategies to detect reporting bias and recommendations to avoid it:

• Specify the primary outcome before performing statistical analysis
• Register study protocols and statistical analysis plans prior to conducting studies
• Use confidence intervals and effect sizes (not p values) to interpret results
• Emphasise study replication and complete publication of research findings

Reference

Albarqouni LN, López-López JA, Higgins JPT (2017) Indirect evidence of reporting biases was found in a survey of medical research studies. J Clin Epidemiol 83:57-64.