This article has Open Peer Review reports available.
Is it becoming harder to secure reviewers for peer review? A test with data from five ecology journals
© The Author(s) 2016
Received: 13 May 2016
Accepted: 28 September 2016
Published: 4 November 2016
The Commentary to this article has been published in Research Integrity and Peer Review 2017 2:3
There is concern in the academic publishing community that it is becoming more difficult to secure reviews for peer-reviewed manuscripts, but much of this concern stems from anecdotal and rhetorical evidence.
We examined the proportion of review requests that led to a completed review over a 6-year period (2009–2015) in a mid-tier biology journal (Molecular Ecology). We also re-analyzed previously published data from four other mid-tier ecology journals (Functional Ecology, Journal of Ecology, Journal of Animal Ecology, and Journal of Applied Ecology), looking at the same proportion over the period 2003 to 2010.
The data from Molecular Ecology showed no significant decrease through time in the proportion of requests that led to a review (proportion in 2009 = 0.47 (95 % CI = 0.43 to 0.52), proportion in 2015 = 0.44 (95 % CI = 0.40 to 0.48)). This proportion did decrease for three of the other ecology journals (changes in proportions from 2003 to 2010 = −0.10, −0.18, and −0.09), while the proportion for the fourth (Functional Ecology) stayed roughly constant (change in proportion = −0.04).
Overall, our data suggest that reviewer agreement rates have probably declined slightly but not to the extent suggested by the anecdotal and rhetorical evidence.
There is a widespread perception among Editors of academic journals, as well as the broader research and scholarly publishing communities, that it is becoming harder and harder to find willing reviewers ([1, 7, 11, 12], Baveye and Trevors 2011 , [8, 9, 13]). The most common reason cited is “reviewer fatigue,” whereby the steady increase in submissions to journals in the past two decades has meant that ever more review requests must be sent to the same limited pool of reviewers [3, 18]. Other factors contributing to fatigue include the falling success rates of funding applications  and the increase in other administrative tasks expected from academics . However, despite the certainty that normally accompanies statements about the increasing difficulty of finding reviewers, there have been very few attempts to quantify the severity of this problem with journal data (for an exception, see ).
The peer review system is consistently ranked by researchers as an important function of journals , . A significant decline in the willingness of researchers to contribute reviews would be a serious problem for scholarly publishing. Even in the short term, a reduced reviewer agreement rate would add significantly to the time it takes to find sufficient reviewers, lengthening the delay between manuscript submission and the editorial decision.
With the exception of Fox et al. , who estimated a reduction of reviewer agreement rates from about 70 % in 2004 to about 45 % in 2014 for Functional Ecology, there have been no published attempts to quantify the reviewer acceptance and completion rates at other journals. Therefore, to add to the Fox et al.  results, we conducted a similar analysis on the rate of review completion in Molecular Ecology from 2009 to 2015. We also re-analyzed publicly available data from four other ecology journals (including Functional Ecology; , [5, 6]) to investigate if there is a larger trend among journals. As a second hypothesis, we also tested whether it was harder to find reviewers for lower quality papers at Molecular Ecology.
Declined: this included reviewers who actively declined as well as those who did not respond to the invitation or subsequent reminders. This response was also used when we could not find a valid email for the reviewer in 2009, 2011, and 2013.
Unassigned: this covered reviewers who accepted an invitation but were later unassigned i.e., they did not complete a review. This response category was included in the total review request count.
Uninvited: invitations that were revoked before a reviewer could respond, typically because the editorial office spotted an error or had sufficient reviews to make a decision. Since these do not allow for an accept/decline choice by the reviewer, these were excluded from the total review request count.
Completed: reviewers who returned a review after accepting an invitation.
The additional category “Invalid Email” was introduced in 2014 to denote review requests that generated an email error. Since this response type was included with “Declined” in previous years, we combined the “Invalid Email” response with “Declined” for the 2015 manuscripts.
Reject: the paper was rejected outright.
Reject, encourage resubmission (REnc): the paper was rejected, but the authors were encouraged to resubmit a new version of the paper for a second round of peer review.
Reconsider after revision (RAR): the paper is provisionally accepted, conditional on the new version adequately addressing requested revisions. The revised version would be re-assessed by the Associate Editor.
Minor revisions (MRev): the paper is provisionally accepted with a request to make some additional changes. The revised version would be assessed by the Managing Editor.
We conducted a logistic regression of the proportion of completed reviews out of all review invitations for Molecular Ecology, with year and editorial decision included as independent variables in the model. Significance of the variables was assessed using likelihood ratio tests that compared the deviance of the model containing the variable vs. the deviance of the model with that variable removed.
Other ecology journals
Petchey et al.  conducted an analysis of the relationship between the number of manuscripts submitted by a researcher and the number of papers they reviewed. They accessed data on articles submitted and review requests sent between 2003 and 2010 for four ecology journals owned by the British Ecological Society (BES): Journal of Animal Ecology, Journal of Ecology, Journal of Applied Ecology, and Functional Ecology. These datasets were collected using ScholarOne Manuscripts reporting system and are available on Dryad (http://datadryad.org/resource/doi:10.5061/dryad.36r69, Petchey et al. 2014b). The “authors” subset of these datasets gives the number of review requests sent and the number of reviews returned for each reviewed submission at each journal, along with the editorial decision. The data included both original submissions and resubmitted manuscripts (Lindsay Haddon, pers. comm.), although these could not be distinguished in the dataset.
Before re-analyzing this dataset, we removed papers where the number of review requests was zero, and the single instance where the number of reviews obtained was higher than the number of requests sent. We did not conduct a logistic regression on the proportion of completed reviews for these journals. This was because the number of papers in each journal in each year was very large (~1000), so that even small differences among years would likely have been statistically significant without necessarily being significant in practice. Therefore, we calculated average proportions per year with 95 % confidence intervals and used those to make comparisons among years. The decision categories at all four journals changed through time and were unclear as to their relative hierarchy. Therefore, we did not use final editorial decision in this comparison.
The results in Fox et al.  are split by gender of the invited reviewer. Since the data for Fox et al.  is publicly available (http://dx.doi.org/10.5061/dryad.5090r, ), we were able to calculate an overall review request acceptance rate that is roughly comparable to the review completion rate for the other journals examined here.
Summary of reviewer invitations and responses
Mean number of reviews per manuscript:
Mean proportion of reviews completed (95 % CI)
0.47 (0.43 to 0.52)
0.44 (0.41 to 0.49)
0.41 (0.38 to 0.45)
0.44 (0.40 to 0.48)
When the Molecular Ecology data are broken down by the editorial decision, there is no clear trend for one type to have received more invitations. Similarly, the manuscripts tended to have a similar proportion of reviews completed regardless of the type of decision (Fig. 2).
Other ecology journals
Our results for the carefully collected dataset in Molecular Ecology found no evidence for a reduction in the proportion of completed reviews between 2009 and 2015. While the number of review invitations seemed to increase slightly over the years, this was counteracted by a larger number of completed reviews over time.
Our results on the re-analyses of data from four other ecology journals suggest that, while current reviewer completion rates are broadly similar across journals, there appears to be a declining trend in completion rates across these four journals through time. For some journals, we also noticed periods of apparent stasis in reviewer completion rate across several years followed by abrupt declines. Reviewer fatigue may have contributed to these declines; however, there are other possible explanations that are addressed below.
Differences between journals and patterns within them could be due to operational and manuscript management reasons. Decreases in the amount of time given for review turnaround may negatively affect completion rates. At Molecular Ecology, the turnaround time for reviews was 2 weeks for the entire period examined here. Journal of Applied Ecology and Journal of Animal Ecology had 3-week review turnaround times, which are fairly generous and these did not change over the period considered (Lindsay Haddon, pers. comm.). At Journal of Ecology and Functional Ecology, there was a switch from 3 to 2 weeks around 2009 (Lindsay Haddon, pers. comm.), so this is unlikely to explain the drop in completion rates in previous years.
Some journals may reject papers outright once a certain number of review invitations are declined, leading to lower completion rates. However, none of the journals examined here have this practice (Lindsay Haddon, pers. comm.). Finally, the number of reminders sent to reviewers may change over time impacting the completion rate. We do not have these data for the BES journals, but at Molecular Ecology reminders changed from two to one in early 2013 and this is not correlated with any change in completion rate (Fig. 1).
Changes in Editor identities (either the Chief Editor or Associate Editors) may affect reviewer willingness to accept or complete reviews. Researchers may be reluctant to accept review requests from particular Editors, and Editors also differ in their skill at selecting people who are likely to accept the review request. However, all journals have Editors of varying reputation and ability, and these differences should average out when data are considered among journals, particularly those with similar scope and impact factor.
Differences in how manuscripts and reviewer responses were coded across years may also contribute to differences between Molecular Ecology and the other journals. For example, since the review process for resubmitted manuscripts tends to be different from original submissions (original reviewers are re-invited; decisions can be made with fewer reviews), it is possible that these trends would be different, or disappear, if these datasets were restricted to original article submissions (as for the Molecular Ecology dataset). The timeline studied was also shorter for Molecular Ecology, and it is possible that examination of a longer time period would have revealed a trend, especially if the rate of decline is slow. Unfortunately, Molecular Ecology only adopted ScholarOne in 2008, and the data from previous years is unavailable.
Interestingly, our results for Functional Ecology do not match those reported in Fox et al.  for this journal over the same period (2004–2010; Fig. 4). Fox et al.  focused on review request acceptance rate rather than review completion rate. They state that approximately 5 % of accepted reviews were not completed but make no mention of whether this rate varies between years. This difference of 5 % would account for the lower estimates of completion rate from the Petchey et al.  data for 2007 to 2010 but not the 10–20 % differences observed between 2004 and 2006. However, Fox et al.  state that non-responses to reviewer requests were not consistently recorded prior to 2007. This may account for the differences since the Petchey et al.  data had consistently higher estimates for the number of review invitations in the pre-2007 years than did the Fox et al. [5, 6] data.
All four other ecology journals show a trending decline in completion rate from 2007 onwards (Fig. 3), and this trend is confirmed to continue to 2014 for Functional Ecology . The discrepancy in the pre-2007 results for Functional Ecology between the two datasets suggest that perhaps the data from all four journals prior to 2007 may not reliably show the proportion of review requests that led to a completed review. The comparison of the Petchey et al.  data with Fox et al.  highlights the possibility that the review request success rates have been steadily declining since 2003 at all four journals.
We did not see a significant decline in review completion rate at Molecular Ecology from 2009 to 2015. These results differ from those at four other ecology journals, where a decline was generally observed. These differences between journals may point to subtleties in the relationship between the effort required to obtain reviews and the specifics of journal identity and management. Overall, our data suggest that reviewer agreement rates have declined at some journals but not to the dramatic extent suggested by the anecdotal and rhetorical evidence. Future research could extend this type of analysis to a broader range of journals, investigate trends over longer time periods, and include more detail about reviewer and editorial characteristics to better control for these factors when making comparisons among journals.
We thank Armando Geraldes and Adrienne Nye for help with the data collection. We are also grateful to Jeremy Fox and Lindsay Haddon for feedback on an earlier version of this paper.
The authors received no funding for this work.
Availability of data and materials
New data from this manuscript will be deposited on Dryad.
JLG and AC collected the data; THV, JLG, and AYKA analyzed the data; and THV, JLG, and AYKA prepared the manuscript. All authors read and approved the final manuscript.
THV, JLG and AC were members of the Molecular Ecology editorial office when the work was conducted.
Consent for publication
Ethics approval and consent to participate
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Anonymous. Peer review—do unto others. Am J Epidemiol. 2010;171:1249.View ArticleGoogle Scholar
- Baveye PC, Trevors JT. How can we encourage peer-reviewing? Water, Air, and Soil Pollution. 2011;214:1–3Google Scholar
- Breuning M, Backstrom J, Brannon J, Gross BI, Widmeier M. Reviewer fatigue? Why scholars decline to review their peers’ work. PS: Political Science and Politics. 2015;48:595–600.Google Scholar
- Canadian Science Publishing. Canadian Researchers’ Publishing Attitudes and Behaviours. 2014; http://www.cdnsciencepub.com/files/PDF/CSP_ResearcherAttitudes_March14_FINAL.pdf. Accessed 13 Apr 2016
- Fox CW, Burns CS, Meyer JA. Editor and reviewer gender influence the peer review process but not peer review outcomes at an ecology journal. Funct Ecol. 2016;30:140–53.View ArticleGoogle Scholar
- Fox CW, Burns CS, Meyer JA. Data from: Editor and reviewer gender influence the peer review process but not peer review outcomes at an ecology journal. Dryad Digital Repository. 2016b; http://dx.doi.org/10.5061/dryad.5090r. Accessed 13 Apr 2016.
- Fox J, Petchey OL. Pubcreds: fixing the peer review process by “privatizing” the reviewer commons. Bull Ecol Soc Am. 2010;91:325–33.View ArticleGoogle Scholar
- Goldman HV. The scarce peer reviewer and challenges journal editors face. 2015; http://www.editage.com/insights/the-scarce-peer-reviewer-and-challenges-journal-editors-face. Accessed 3 Apr 2016.
- Grossman GD. Improving the reviewing process in ecology and evolutionary biology. Anim Biodivers Conserv. 2014;37:101–5.Google Scholar
- Kinman G, Wray S. Taking its toll: rising stress levels in further education: UCU Stress Survey 2014. 2015; https://www.ucu.org.uk/media/7265/UCU-FE-stress-survey-2014---report/pdf/ucu_festressreport14.pdf. Accessed 13 Apr 2016.
- Lajtha K, Baveye PC. How should we deal with the growing peer‐review problem? Biogeochemistry. 2010;101:1–3.View ArticleGoogle Scholar
- Martin BR, Bell M, Callon M, Grupp H, Kodama F, Kuhlmann S, et al. EES and the continuing evolution of research policy. Res Policy. 2009;38:695–9.View ArticleGoogle Scholar
- Merrill E, Cox A. Reviewer overload and what can we do about it. J Wildl Manag. 2014;78:961–2.View ArticleGoogle Scholar
- NIH Databook. 2015; http://report.nih.gov/NIHDatabook/Charts/Default.aspx?showm=Y&chartId=275&catId=13. Accessed 13 Apr 2016
- Nature Publishing Group. Author Insights 2014. 2015; https://dx.doi.org/10.6084/m9.figshare.1204999.v4. Accessed 20 Apr 2016.
- Petchey OL, Fox JW, Haddon L. Imbalance in individual researcher’s peer review activities quantified for four British Ecological Society journals, 2003-2010. PLoS One. 2014;9:e92896.View ArticleGoogle Scholar
- Petchey OL, Fox JW, Haddon L. Data from: Imbalance in individual researcher’s peer review activities quantified for four British Ecological Society Journals, 2003-2010. Dryad Digital Repository. 2014b; http://dx.doi.org/10.5061/dryad.36r69. Accessed 13th April 2016.
- Siegel D, Baveye P. Battling the paper glut. Science. 2010;329:1466.View ArticleGoogle Scholar
- Vines T, Rieseberg L, Smith H. No crisis in supply of peer reviewers. Nature. 2010;468:1041.View ArticleGoogle Scholar