Open Science Collaboration Authors on the Rate of Reproducibility in Psychological Science
In the second bimonthly update of Essential Science Indicators from Clarivate Analytics, the paper “Estimating the reproducibility of psychological science” (Science 349 [6251] 28 August 2015), was named both a New Hot Paper and a Fast-Breaking Paper for Psychiatry & Psychology. Currently, in the Web of Science, this paper has 479 total citations since its 2015 publication.
Below, two of the authors of the paper, Johanna Cohoon and Mallory Kidwell, share a behind-the-scenes look at the development of this work and its implications for the field of Psychiatry & Psychology.
Would you summarize the significance of your paper in layman’s terms?
Our paper attempted to conduct replications of 100 research findings from articles published in leading psychology journals. We did this so that we could estimate of how often such findings could be reproduced by psychologists outside the original research team. We found that when following the methods described in journal articles, supplemented by original authors’ recommendations, it is extremely difficult to achieve the same results as originally reported. When the reproducibility of these 100 findings was evaluated across several measures, including p-value, effect size, and replication of authors’ subjective assessment, we found that less than half of the findings could be successfully replicated.
A failure to replicate has four distinct and equally plausible explanations:
- The original effect may have been a “false positive”—meaning that the original study’s finding was false, and the replication’s test of the effect was correct. This is the most commonly assumed implication when reading our findings; however, our paper strongly argues that this is not the only possible explanation.
- An alternative explanation is that the replication effect may have been a “false negative”—that is, the replication’s finding was false and the original study’s test of the effect was correct.
- Another alternative is that both the original and replication findings are accurate, but the methodologies employed differed in an important way—we attempted to reduce the impact of this explanation by including the original authors in the development of the replication protocol, but this explanation is still possible.
- Finally, it is possible that the phenomenon under study is not well known enough to anticipate differences in the sample or the environment—that is, we may not know enough about the phenomenon we are trying to test in order to measure or manipulate it correctly.
It is important to note that we cannot be sure which explanation is correct based on the results of one replication alone. This has many implications for how scientists interpret the research they read, conduct research in their labs, and extend research from previously published work. Regardless of the reason, our work shows the importance of thoroughly documenting the scientific process so that others may better evaluate it.
Why do you think your paper is highly cited?
Reproducibility in science, let alone psychology, is not a new topic of interest. It is often central to arguments for open science—the idea that research process and findings should be as transparent and easily accessible as possible. The spotlight over the open science movement has been growing over the past decade. Our article added to this ongoing discussion by providing necessary empirical evidence of the rate of reproducibility in psychological science.
Our findings were hotly debated when we published our article, drawing still more attention to the issue. Some saw our research as a clear indication that the way scientists conduct research needs an overhaul. Others argued that we should have approached the problem differently. Such debate is common and healthy but because of the alarming reproducibility rate we calculated and its implications, many researchers chimed in to either support or dispute our claims. We framed our findings so that they could be understood and applied in other fields as well, meaning that researchers outside of psychology and within its many subdisciplines could participate in the discussion.
Outside of the useful evidence we provided and the contentious nature of our findings, another reason our article may be popular is because of our approach to the research problem. We collaborated with about 300 other researchers. This kind of large-scale collaboration is not common in psychology. Additionally, we attempted to conduct the project as transparently as possible; we shared our protocols, drafts, materials, and data with anyone who visited the Reproducibility Project: Psychology webpage. Other researchers may cite our article not just because of the findings, but because it exemplifies and details a new open and collaborative approach to research in psychology.
Does it describe a new discovery, methodology, or synthesis of knowledge?
Our paper actually describes a new discovery using new methodology. No significant attempt to empirically study the rate of reproducibility in psychology had been attempted before. We were able to arrive at an evidence-based result by conducting a meta-analysis of the 100 replications conducted by our co-authors. Our findings were the first to truly quantify how difficult it is to replicate research in psychology.
The methods we used were also novel in the field of psychology. Our collaboration with hundreds of co-authors—an uncommonly large collaboration in psychology—required significant coordination and documentation. That work provided many “lessons learned” that other scientists may benefit from. Additionally, our approach provided a blueprint for how an “open” project might look. Open science, while not new, is a growing movement that can benefit from such concrete examples of how to practice transparent research.
How did you become involved in this research, and how would you describe the particular challenges, setbacks, and successes that you’ve encountered along the way?
Researchers at the Center for Open Science (COS), a non-profit dedicated to the openness, integrity, and reproducibility of all scientific research, coordinated the project from its inception in 2011. While there was substantial debate among the scientific community about whether reproducibility of findings was an issue, there was little evidence to support either argument. We joined the team in 2013 (Johanna) and 2014 (Mallory) to see this project to completion and help realize COS’ mission with a tangible product. To do so, we recruited researchers from across the world to join us in the endeavor of collecting data to test psychology’s reproducibility directly.
Ultimately, we worked with over 300 authors and volunteers to conduct 100 replications, analyze the results, and publish the meta-analysis that details our findings. However, we found that we learned just as much from the process of coordinating this project as we did from the data and results. Coordinating 300+ researchers with busy schedules outside of this project was difficult, and one of the main challenges we encountered was ensuring that all teams made the right amount of progress and met project deadlines at the appropriate time.
Along the way, we also learned firsthand the importance of extensive documentation (both of the original and replication studies) and access to the original materials administered to ensuring a close replication. When details and materials for the original were available it was much easier to create a replication that mirrored the original in every aspect. Accordingly, we strove to provide thorough documentation for our replication studies to improve the accessibility and transparency of our own methods from the moment we began recruiting the first research teams. This meant viewing each step in our research through the eyes of a stranger—we had to consider what a newcomer would see and understand if they were to view our documentation for the first time. The challenges and experiences we faced in the process of implementing this project, as well as the ultimate results of our efforts, have important implications for how researchers can improve their scientific method.
Where do you see your research leading in the future?
Our research furthered the discussion of reproducibility in science by providing an estimate of how often original findings in psychology may be reproduced. While the principles of our findings are relevant to all disciplines, our empirical results only apply directly to one field. Other projects, such as the Reproducibility Project: Cancer Biology, will provide similar estimates in other fields. Other researchers have also studied reproducibility from different angles, with mathematical models of reproducibility and other commentary on how to improve documentation and transparency in research. Hopefully, this work will continue until we have a strong evidence for the factors that promote or hinder reproducibility. As we work toward that understanding, we can continue to improve scientific research practices using the evidence we have to date.
Additionally, because our project was conducted in an open, transparent manner, we have shared our dataset on the project’s website. This means that any researcher can replicate our own analyses or use our data to answer new questions that any scientist might pose.
Do you foresee any social or political implications for your research?
Our results can be used to justify policy changes that support open science and other measures to improve reproducibility. Journals, for example, may raise their documentation expectations. Authors might be asked—and are asked when submitting to some journals—for their data, analysis scripts, or supplementary materials to provide more documentation than a traditional journal article affords. This extra information can help readers evaluate the science and can greatly assist in conducting replications.
Funders may also raise the bar for documentation and their expectations of transparency. Most publicly funded research is currently subject to rules that are meant to encourage sharing and transparency. Too often, however, those rules go unenforced or are too vague to effect much change. Being more specific about what grant awardees are expected to share can help ensure that original materials and data—which can be vital to reproducibility—are available.
Our research has added to a greater conversation about transparency and openness in the sciences. We hope that our evidence has provided an impetus for policy action and that it will inspire future work that helps us better understand reproducibility and what open science can offer researchers.
Johanna Cohoon
Doctoral Student, School of Information
University of Texas at Austin
Austin, TX, USA
Mallory Kidwell
Doctoral Student, Clinical Psychology
University of Utah
Salt Lake City, UT, USA
