It’s not the size that matters

We celebrate 300K reviewers on Publons with a new data project: join us as we uncover the average length of scientific peer review across disciplines, journals, and countries, and question what it means for review quality.

One of the most important criticisms of the peer review system is that it is much too slow. It can take upwards of 180 days to publish research–sometimes more.

But there’s another criticism lurking in peer review circles that few dare to talk about. Size.

How long is, you know… your peer review? And what is average, anyway?

We wanted to find out. And seeing as 300,000 researchers are now getting recognition for their peer review and editorial work on Publons…

We thought it was perfect timing to say thank you for being part of our journey so far, and to share some insights with you all!

So! Is the average peer review short and blunt, fair to middling, or long and verbose?

Before we answer that, it’s important to remember that review length can depend on several factors…

The most obvious of these factors is the quality of the manuscript. Researchers will typically spend around five hours reading and reviewing a paper, but that can vary depending on the complexity of the topic, the potential of the manuscript, and the experience of the reviewer. Anecdotally, reviewers report spending less time on papers that are very poor, or conversely, very good.

Time is also a big factor. Today’s “publish or perish” culture has placed unrealistic demands on researchers’ time, most of which is geared towards the publication of research rather than service to their field or profession. Greater recognition for peer review can be instrumental in changing this — This is something we’ve written about a lot in the past and makes up a bit chunk of our Editors’ Guide to Finding Reliable Peer Reviewers.

The researcher’s mood, the manuscript’s relevance, and the amount of peer review training they have had can also play a part in what ultimately translates into review length, and of course, review quality.

But there are other, more subtle reasons why your peer review takes its final shape. We dug deep into the Publons database to find out what these trends may be and uncovered some interesting results relating to geography, field, Journal Impact Factor–and even the editor soliciting the review.

(By the way, if you’re already on Publons, don’t forget you can see your average review length here.)

Here’s what we found…

The total number of reviews we analyzed was 378,561. We’ve presented our findings as notched box and whisker graphs, with the median and upper and lower quartiles shown. An estimate of the 95% confidence interval of the median is provided by the range of the notch. The whisker portions encompass 99% of the data. Anything outside of this range is shown as a marker. You can read more about this and our methods in the Appendix.

Review length by country (top 20 countries by review count)

First up, we looked at review length alongside country data. Of the 378,561 reviews we analyzed, 233,283 had country data associated with them.

The results of this graph are interesting. It looks as if reviewers in countries that are English-speaking or have a high proportion of English speakers (European countries, for example), tend to write longer reviews than their peers in Asian and Middle Eastern Countries.

 

Review length by journal’s top Web of Science subject category

Of the 378,561 reviews we analyzed, 319,736 were associated with a Web of Science Subject Category. We grouped the categories into higher-level fields to analyze the way review length varied by broad disciplines. It must be noted that any review for a journal that has more than one Web of Science subject category was included in the fields for both–or all–categories, which is the reason a higher percentage of reviews are counted towards commonly associated fields (Biology and Medicine, for example).

It would seem that reviews in disciplines where the articles tend to be more discussion-based, (Psychology, Environmental Sciences, Arts and Humanities, for example) are longer than fields where the articles would be more like proofs (e.g. Mathematics), or fields where research articles often appear as pre-prints (e.g. Physics).

 

Review length by Journal Impact Factor

 

Here’s where things get really interesting.

Of the 378,561 reviews we analyzed, 329,109 were for journals with a Journal Impact Factor. Each ventile contains approximately 16,500 reviews.

As you can see from the graph there is a clear shift towards longer reviews as the Journal Impact Factor increases. This may not be surprising. The Impact Factor ranks journals according to the average number of citations that their articles receive over two years. Researchers hoping to secure publication spots in these journals will likely consider a high-quality review as a form of investment in their future prestige.

Another option is that editors of journals with higher Journal Impact Factors solicit more reviews from European countries (who we’ve seen tend to write longer reviews). We decided to investigate this further in the plot below.

 

Review length and Journal Impact Factor by reviewer country

The relationship between the median review length and median Journal Impact Factor by reviewer country is interesting.

Reviewers from “western” countries tend to write longer reviews on average when it’s associated with a high Journal Impact Factor. Reviewers from Japan, China, and Brazil tend to write shorter reviews for mid-range Impact Factors, and reviewers from India, Iran, Turkey, and Egypt tend to write shorter reviews for journals with lower Journal Impact Factors.

This is suggestive that reviews for journals with a higher Journal Impact Factor are predominantly done by reviewers from (or in, depending on how they classified themselves) western countries.

We delve a bit more into editor’s global search for reviewers. It doesn’t touch on the Journal Impact Factor, but it’s an interesting take on how far afield editors look for peer review support.

 

Predictions for the future

It will be interesting to see how these results change over the next few decades. There are indications that review length is getting shorter: in 2016 the average word length on Publons was 457 words and in 2017 we saw a 23% drop to 342.

However, while that is significant, it must also be considered against the massive growth we had on Publons over that period. Both the number of reviewers who joined and the number of reviews they added on Publons more than doubled between 2016 and 2017.

It must also be considered against our own “selection bias” — that is, those numbers are restricted to Publons data and come at a time when gaining recognition for peer review was still relatively new. Early adopters to our network were likely to understand the importance of peer review and spend a considerable amount of time and effort on their contributions. Our latest data represent a much larger part of the population and may now be closer to demonstrating the community’s trends in peer review more generally.

Our prediction is review length will increase on average as we see a renewed focus on improving the quality of peer review.

More recognition for peer review will determine the time and effort researchers can (and are willing) to put into review. This ultimately comes down to institutions and funding bodies acknowledging peer review contributions as an indicator of a researcher’s expertise and an aspect (or facet) of their impact in their field.

This is starting to happen: more and more researchers are telling us how peer review is gaining front-row status in job and funding applications. The latest of which was
Amanda Salis from the University of Sydney. In 2017 she received both a promotion and five years’ worth of funding for her research:

 

“I have observed a change of culture at the National Health and Medical Research Council and at research institutes in the past 5-10 years that has encouraged the addition of peer review activity to research applications. It is no longer enough to just be doing great research; you also need to demonstrate that you are contributing to your profession more broadly.”

 

As more institutions and funding bodies start recognizing the importance of peer review, the more time researchers will contribute towards it, and the better and faster the entire publishing process will be.

There are also more peer review training courses available for new and experienced academics, and that will help inform a greater number of researchers involved in the task. The Publons Academy, for example, offers short online (and free!) video modules on all the core components of review.

This has helped early career researchers like Edmond Sanganyado submit critical comments on the novelty, significance, and quality of research rather than short notes on the language used. You can read about his journey through different peer review courses and his 5 tips on writing a structured review report.

The growing popularity of preprint servers is also likely to impact the quality of review in the future, and this may also result in longer reviews. The number of preprints is rapidly expanding as funders shift towards a culture of openness. This is great for transparency and even better for peer review. If research is made more readily available through preprint servers then peer reviewers ultimately have more time to craft better — and perhaps longer — peer reviews.

So with all of this said, can we say with certainty that a good review is a long review?

Like many hot topics in the research community, it is difficult to say for sure. We approached this question by looking at geography, subject field, and Journal Impact Factor, and while we uncovered some interesting trends we cannot give any concrete answer on whether longer reviews are better or worse than shorter reviews.

What we can say for sure, however, is that a peer review can be as long as a piece of string because it’s not the size that matters.

Great reviews can be short and concise. Poor reviews can be long and in-depth, or vice-versa.
What counts is the quality of the work submitted, and that’s where we need to focus our efforts. Peer review can be practiced, it can be improved, and it should be recognized as the important skill it is. It’s on all of us to do what we can to work towards that.

So, what do you think? Share your thoughts on our findings and let us know what your average review length is and why.

Special thanks to Joseph Corbett and Alex Sokolov for putting all of this data together.

 

Raw data

You can access the raw data we used in this blog here. Please note that we have excluded any review information if there is <2 reviewers per country. Reviews appear under several fields if the journal has more than one top subject category, and reviews can be grouped by review_hash column which is a hashed value of review identification.

 

Appendix

The total number of reviews we looked at was 378,561. These were all first-round reviews with a word length >1. To carry out the analysis we simply split the reviews into words and counted the words — it’s important to note that no quality control of the reviews was performed.

The limits on the y-axis were set to 3000, although there are a small number of reviews whose word counts are greater than 3000. The mean values are also included, but caution should be used in interpreting these as the distributions are not Normal.