Openness and transparency are essential to creating trust in the peer review process. With research integrity being the theme of this year’s Peer Review Week, we look at ongoing efforts that Clarivate™ is conducting, in collaboration with colleagues in the scientometric research community, to further open peer review. We also look at how far open peer review has come and what the future holds.
Peer review is an essential part of the scientific endeavor yet challenges exist throughout.
In its essence, peer review provides a checkpoint where research results are scrutinized by experts in the field, who in turn, provide their opinion on the validity and quality of the research. Peer review is a part of the service provided by researchers to their peers and is normally unpaid. However, with the exponential growth in research publications, journals are struggling to find appropriate reviewers for submitted papers. Since in-depth peer review is time consuming and requires attention to all aspects of a paper, many researchers cannot keep up with the growing number of papers they are asked to review.
This challenge, in and of itself, creates a fertile ground for authors and publishing outlets to skimp on the rigorous process of peer review.
Significantly cutting the number of reviewers, publishing poorly reviewed articles, or in certain cases faking the entire review process are some of the phenomena we witnessed in the past few years.
According to the Retraction Watch database, over 3,000 articles have been retracted due to fake peer review. Fake peer review can involve submitting fictional or real names of reviewers, with false email addresses that return to the authors or to unscrupulous editorial firms. New examples of this are occurring frequently, with PLOS ONE announcing in March it will be retracting more than 100 articles for manipulated peer review.
Such scandals reveal vulnerabilities in the peer review system and are certainly a growing problem for editors, publishers, institutions and database providers who are battling this phenomenon to ensure research integrity in science.
How open peer review addresses challenges in the peer review system
Open peer review, also known as transparent peer review, refers to the practice of publishing review reports and sometimes also the identities of reviewers. In recent years, there has been a large growth in the adoption of open peer review. The transparency offered by open peer review ensures that the quality of peer review, and also the quality of the paper under review, is made visible for readers of the paper. This enables anyone who considers building upon the paper to develop an understanding of strengths and weaknesses of the research.
Open and transparent peer review is not new. Publishers like Journal of Medical Internet Research, British Medical Journal, and BMC have put the concept into practice for over 20 years. While many publishers have continued to innovate in this area since then, widescale and cross-disciplinary adoption has been lagging. This is often due to real and perceived concerns about researcher attitudes towards the practice. There is also the complexities of building on top of existing journal technologies and workflows.
Transparent Peer Review
In 2018, Clarivate launched a new publisher service called Transparent Peer Review (TPR). This innovative solution brings openness to peer review by exposing a comprehensive peer review history including reviewer reports, editor decision letters and authors’ responses, all linked from the published article. Numerous studies and surveys over the last 10 years show that researcher attitudes toward open and transparent peer review, while still varied, are becoming more open to the practice. In 2018, Clarivate piloted TPR with Wiley to tackle the technical and workflow complexities, with a focus on:
- Providing a robust path to transparent peer review for any journal
- Requiring minimal work for a publisher to set-up transparent peer review
- Making the process as simple as possible for authors, reviewers and editors.
Publishers using ScholarOne™ as their journal submission system can use the TPR system. TPR originated with Publons™, now fully integrated with the Web of Science, making peer review more visible and giving researchers recognition for their peer review activities. TPR has grown over the last three years and is currently used by 123 journals across 7 publishers, and TPRs are fully available in Web of Science for both subscribers and non-subscribers. See for example:
Web of Science Core Collection™ record with open peer review data:
Open peer review details:
And here is the full Web of Science record for this publication.
Usage of open peer review
We recently examined the usage of TPR to identify trends. After all, transparency is great, but are scholars actually viewing the transparent reviews? As of July 2022, TPR contained 19,277 articles: 17,986 with citation information (6,514 had zero citations), 19,214 with peer review report (PR) view information, of which 496 had zero PR views. The first paper in TPR was published in January 2018. Among those with TPR views, six papers accrued more than 1,000 PR views and 243 papers 100 or more views. Three papers received more than 100 monthly average views and 419 papers received 10 or more monthly average views.
The distribution of citation counts and peer review report views is shown below: The box plot presents the log-normalized distribution of citation counts (purple) and TPR view counts (blue) for the 19,277 papers with peer review reports available. The median TPR view count is 17, and the median citation count is 6. Overall, articles receive more TPR views than citations.
Figure 1: Distribution of citations and views by article
Source: Web of Science
The box plot shows the log normalized distribution of citation counts (purple) and TPR view counts (blue) for the 19,277 papers with peer review reports available. The median TPR view is 17, and the median citation count is 6. The lower rates for citation is expected, given it takes a longer time for papers citing these papers to get published. About 20% of total articles account for 52% of TPR views, and 49% of total articles account for 80% of total TPR views in the dataset. A similar skewed pattern is also commonly observed with citations.
Figure 2: Cumulative distribution of Transparent Peer Review articles and views
Source: Web of Science
We also looked at the data by publisher. Wiley, the pilot publisher for TPR, is the largest provider of TPR articles to date, in terms of both the number of articles and journals. There is only one journal by MIT Press (Quantitative Studies of Science). We calculated views per article to normalize for the large size difference among the publishers’ TPR articles. MIT Press is also the publisher with the highest number of PR viewers per article (48.77).
Statistics by publisher:
Source: Web of Science
To understand usage by the community, we analyzed the monthly page views for TPR articles from 2020 to May 2022.
Figure 3: Transparent Peer Review page views by time
Source: Web of Science
The trend of TPR articles published, along with average views per article, is shown below. Overall, the total number of TPR views is increasing as more TPR papers are added from only 15 papers in 2018 to more than 9,000 papers in 2021, while average views per article has decreased with this growth.
Figure 4: The trend of TPR articles published, along with average TPR views per article
Source: Web of Science
Publish Your Reviews initiative
Open or transparent peer review typically refers to the publication of reviews alongside articles published in a journal. In recent years, however, researchers have also started to publish reviews of preprints, partly to address the need for rapid peer review of COVID-19 research. A next step taken by some researchers, including one of the authors of this blog (Ludo Waltman), is to publish journal-commissioned reviews alongside the preprint version of an article. This new form of open peer review is promoted by the Publish Your Reviews initiative, launched recently by ASAPbio with support from a large number of organizations. Publication of journal-commissioned reviews has several benefits. It for instance offers valuable contextual information to readers of preprints, and it enables journals to reuse each other’s reviews.
What’s next for open peer review?
Near term, Clarivate is focused on opening Transparent Peer Review to any of the 8,500+ journals using ScholarOne. This includes standard options for TPR workflows as well as publishing additional peer review objects, such as files from the reviewer’s report or author’s response to decision letter. Clarivate is also exploring how open peer review content from other peer review and preprint systems can flow into the Web of Science, alongside the published article. The vision is for peer reviews to become a standard and discoverable part of the scholarly record across all disciplines. Now that peer review records are visible within the Web of Science, for both subscribers and non-subscribers, we expect to see more usage and engagement.
Thank you to all the peer reviewers
Peer review is a longstanding practice that is essential to scholarly publishing. It is the gold standard for ensuring quality and integrity in research and yet is almost entirely reliant on the voluntary contributions from researchers. Peer Review Week is about recognizing these reviewers and we want to thank everyone involved in peer review for the essential role you play in maintaining research quality.
Learn more
Publishers interested in implementing Transparent Peer Review for their journals can discover more about this innovative service. Researchers and students seeking education on conducting high-quality, responsible peer reviews can use Web of Science Academy, a source for free, online courses on the peer review process, as well as ethical publication and citation behavior. The course contains 7 modules that cover various aspects of peer review including specific attention to reviewing papers in a variety of disciplines.
Contributors
Thank you to all the co-authors of this blog, including:
Ludo Waltman, Professor, Centre for Science and Technology Studies (CWTS), Leiden University.
Erjia Yan, Associate Professor, College of Computing and Informatics, Drexel University.
Chaoqun Ni, Assistant Professor, Information School, University of Wisconsin-Madison.
Ann Beynon, Strategic Alliances and Engagement Manager, Institute for Scientific Information, Clarivate.
Josh Dahl, Product Director, Clarivate.
