Eight Things You Should Know About the Reproducibility Crisis

A 2016 nature survey showed that more than two-thirds of published academic research is not replicable. This is a benchmark of scientific quality, and papers are supposed to detail instructions for other researchers seeking to replicate their results. In fact, researchers are supposed to ensure they can replicate their own results before submission to journals.

Despite this, the vast majority of modern research findings cannot be reproduced by other researchers. Every area of research feels the effects of this, because time and money can’t be directed at only the most promising research.

Here are eight things you should know about the reproducibility crisis:

Fraud is not a major issue: Though there have been a couple of cases of fraud, usually the lack of replicability is not due to fraud.[i] Instead, it’s often due to corner-cutting and poor scientific practice.
Researchers can help by sensationalizing less: Overstating or otherwise enhancing results is a major cause of the reproducibility crisis.[ii] Researchers and publishers like to publish the most interesting results possible, but good science isn’t always eye-catching.
Open research can help: The open access movement encourages publication at all phases of the research process, rather than just at the end.[iii] This increases data quality and quantity, and allows more checks throughout the process. It also decreases the incentive to publish sensational results, meaning the final result will be the highest possible quality. Even traditional publishers are starting to require the publication of open access raw data, which can have a similar effect on reproducibility.
Universities can help by loosening incentive structures: “Publish or perish” increases tendencies to publish too quickly, resulting in lower-quality, incomplete research.[iv] It can also increase the tendency to sensationalize results for publication.
Sometimes you learn more when a study can’t be replicated: Discovering the reason a study couldn’t be replicated has led to some scientific discoveries.[v] There is still a crisis, but not every irreproducible study is destructive.
Most researchers don’t think lack of reproducibility necessarily means a result is wrong: In the same survey that prompted discussion of the crisis, less than 31 percent of researchers said they thought a failure to reproduce results meant they were wrong.[vi] The issue, they said, was more of minimizing false leads and discovering where the disconnect lay.
Journals can address this using statistics rules: A study on reproducibility in psychology journals showed that when the journals included recommendations on the responsible use of statistics, the quality of research improved.[vii]
Rules for replicating studies haven’t changed: Researchers interested in replicating a study should still primarily look at whether there would be value in replication, and whether they think the original methodology was sound.[viii] Researchers should always publish their replication attempts — both successes and failures — even if only in their institutional repository. This will help minimize wasted research time and resources.