Researchers are increasingly encouraged to implement reproducible research practices in their work. These practices include describing the data collected and used for analysis in detail, clearly reporting the analysis method and results, and sharing the dataset and statistical or analysis code. To determine how well reproducible research practices are implemented, Page and colleagues (2017) investigated their implementation in systematic reviews (SRs) of biomedical interventions.

Investigators retrieved a random sample of 300 SRs, and extracted data on epidemiological and reporting characteristics from 110 SRs of randomised trials or non-randomised studies of interventions which reported at least one meta-analysis. Data on items that characterised reproducible research practices were extracted. For example, data were collected on whether the study:

• Reported the data needed to recreate all meta-analytic effect estimates in the SR
• Reported the data needed to recreate the index (first reported) meta-analytic effect estimate
• Reported summary statistics for each individual study in the index meta-analysis
• Reported effect estimates and measures of precision for each individual study in the index meta-analysis
• Reported whether some data in the index meta-analysis had been imputed, algebraically manipulated, or obtained from the study author
• Reported the type of random-effects method used for the index meta-analysis (e.g. was the between-trial variance estimator stated or inferred)
• Mentioned access to datasets and statistical analysis code used to perform analyses

What did they find? 78 (71%) of the sample of 110 SRs of interventions were non-Cochrane SRs. Only 72 (65%) of systematic reviews reported data needed to recreate all meta-analytic effects. This proportion was higher in Cochrane than non-Cochrane SRs. Systematic reviews that reported imputing, algebraically manipulating, or obtaining data from the study author did not always specify clearly which data had been handled in a particular way. Only 33 (30%) of SRs mentioned access to datasets and statistical analysis code.

Investigators concluded that reproducible research practices are underused in systematic reviews of biomedical interventions. They encourage reviewers to adopt and implement these research practices to allow easier identification of errors and to make research transparent.

Reference

Page MJ, Altman DG, Shamseer L, McKenzie JE, Ahmadzai N, Wolfe D, Yazdi F, Catalá-López F, Tricco AC, Moher D, Reproducible research practices are underused in systematic reviews of biomedical interventions, Journal of Clinical Epidemiology (2017), doi: 10.1016/ j.jclinepi.2017.10.017.