A) Set your open peer review goal(s)
A1. Decide what you would like to achieve with OPR
Any journal editor or publisher wishing to implement some form of OPR would be well-advised to first do their homework. What do you want to achieve? How? For which reasons? Answering these questions first will enable you to orient your engagement with OPR. Examine which particular aspects of your peer review processes you would like to improve. For example, do you want to increase the transparency of your processes, give credit to peer reviewers, enable greater participation, or just speed up the peer review process? Being clear on these primary goals is vital.
A2. Acquaint yourself with the differences between the elements of OPR
As discussed above, ‘open peer review’ can mean different things to different people. As a first step, familiarise yourself with the differences between each of these elements. For example, one of the current authors created a taxonomy of seven core traits:
-
1.
Open identities: authors and reviewers are aware of each other’s identity.
-
2.
Open reports: review reports are published alongside the relevant article.
-
3.
Open participation: the wider community are able to contribute to the review process.
-
4.
Open interaction: direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged.
-
5.
Open pre-review manuscripts: manuscripts are made immediately available (e.g., via preprint servers like arXiv) in advance of formal peer review procedures.
-
6.
Open final version commenting: review or commenting on final ‘version of record’ publications.
-
7.
Open platforms (‘decoupled review’): review is facilitated by a different organisational entity than the venue of publication.
Read widely to familiarise yourself with the pros and cons of each of these elements (a primer with links to some literature is given above in the ‘What issues with peer review does open peer review address?’ section and a longer list of secondary reading is included in the Additional file 1).
A3. Decide which elements you would like to implement
Being clear on your primary goals and relating them to specific elements of OPR will enable you to begin to build a provisional strategic plan for OPR implementation. Further refine this by studying existing models and OPR implementations through publisher websites, published literature, presentations and online resources. Use industry contacts and discussion platforms to discuss and learn lessons from publishers and journals who have already implemented OPR procedures. Be aware that the resource and time commitment is dependent on the elements selected—disclosing names of reviewers is relatively straightforward whereas publishing a full peer review history for each paper requires significant investment (see also the ‘Assess technological feasibility of various options’ and ‘Assess the costs of various options’ sections below).
B) Listen to research communities
B1. Be conscious of, and sensitive to, community differences
Be conscious that there will be differences in perceptions and willingness among different research communities. For instance, some disciplines have more of a tradition with double- or even triple-blind review and this might lead to more resistance to openness in peer review. If you are a publisher overseeing peer review at many journals, consider starting with particular disciplines that are more open to trial OPR, especially those where other journals in the field already use OPR (although note that this may be challenging for broad scope interdisciplinary journals).
B2. Consider surveying community opinions
Consider directly surveying community opinions regarding open peer review models to gauge attitudes. This may work especially well for journals with close-knit communities—for example, society journals, which regularly seek feedback from authors or society members regarding journal policies. Alternatively, or as a complement to this strategy, consider targeted ‘qualitative interviews’ to gather insights from those with particularly strong opinions regarding open peer review.
B3. Communicate your goal with the stakeholders and research community
Engage journal communities—firstly by consulting your editorial board and reviewers to get them on board with the idea. It may be necessary to ‘sell’ the benefits of opening up peer review and provide reassurances. A committed and engaged Editor who can drive such discussions may help here. Find keen researchers to work with and gauge interest in the model among communities the journal serves. Let reviewers, authors and readers know in advance, and if you are unsure of how such developments might be received, consider announcing plans in a journal editorial and seeking community feedback. In any case, include requests for community feedback in any such announcements to ensure alignment with researcher attitudes.
C) Plan technologies and costs
C1. Assess technological feasibility of various options
A deciding factor in your prioritising the elements of openness to include will be the technical possibilities of your system. Whether you are a small publisher using open source software or a large publisher which uses one of the major manuscript handling services, if your electronic editorial office and production/publication systems and workflows cannot currently be easily configured for OPR elements, they may be difficult and/or expensive to implement.
C2. Assess the costs of various options
It is important to recognise potential costs in advance. As things stand, there is a lack of infrastructure to facilitate automated workflows for many of the elements of OPR. Hence, development costs may be a major barrier—especially for smaller players. Ask yourself: Which options does your system already support, and do you have the technical staff or resources to fund system development? Consider also that costs will likely not only be in initial implementation (e.g., custom system development), but also ongoing support costs (e.g., staffing). If your needs would require significant custom system adaptations from a third-party service provider, you might consider partnering with other publishers who use these services to spread costs in implementing these changes. Alternatively, some platforms are now offering specific OPR functionality to work together with more traditional publishing services. In any case, be aware that there will usually be different ways, with differing levels of elegance and cost, to implement OPR options. For instance, publication of peer review information could be as simple as manually compiling review components and publishing a single document as Additional file 1 or as complex as an automated (XML) workflow where each element is published separately (see also the ‘Open reports’ section).
C3. Consider workaround options for piloting
If you are just experimenting with OPR, it may be that rather than immediately extending your whole publication architecture, it might be better to start small with workarounds, although be aware that ad hoc workarounds may produce a less smooth user-experience which could affect uptake and user attitudes to the experiments. Consider, however, that the sub-optimal nature of workaround solutions may then become an inhibiting factor in the success of the experiment. One solution here would be for a third-party OPR platform to offer their service as a plug-in to existing workflows for conducting such experiments.
D) Be pragmatic in your approach
D1. Set priorities and consider a phased approach
Be flexible and choose your battles carefully. Change is difficult and you may run into problems if you try too many things at once. Your communities may be more receptive to some elements than others, and so, prioritising the areas you would like to change and being prepared to compromise from the ideal situation or at least take a phased approach may help you maintain traction and community buy-in. It will also make it easier to systematically assess the success or otherwise of any particular innovation.
D2. Consider making options optional or piloting them first
For elements you would like to introduce but think might prove controversial, you could make them optional. Thereby, it is possible to signal your support for this innovation while allowing reviewers or authors to opt-out. Note, however, that default policies may significantly affect outcomes—if the default policy is opt-in, this might lead to lower participation than if the default were to opt-out, for instance. If reactions among research communities may be uncertain, consider introducing OPR through a pilot study with an accompanying survey for participants which would show that any final decisions would be based upon real experiences, whilst allowing the journal to experiment with the confidence of the community.
E) Further communicate the concept
E1. Engage the community, especially via ‘open champions’
Once you have decided on the model you’d like to move to, you have your communities on board, and have prioritised which OPR elements to implement, you will still need to sell your communities on the concept. As a general strategy, you should engage with the research community to find academics who are enthusiastic about OPR to be ‘open champions’ in advocating to their peers—for example, by engaging people who responded positively to your initial community consultation in step B. Moreover, the arguments above in favour of the various aspects of OPR will help sell the concept, especially with regard to increasing transparency, enhancing credit for review activities and demonstrating and (although this is an understudied area) potentially enhancing the quality of reviews.
E2. Be aware that communication is key and terminology is important
Misunderstandings could derail processes. As the stewards of the peer review process, publishers and editors have a duty of care to ensure reviewers and authors fully understand the systems of peer review in which they participate and its potential advantages and disadvantages. Use editorials, webinars, infographics and/or blog posts to articulate decisions and justify why these decisions have been made. Formulate clear policies which are easily findable on journal webpages for authors and reviewers.
F) Evaluate performance
F1. Have a clear framework for assessing success
There is a need to track review quality and acceptance rates to monitor how OPR affects processes. As said above, it is good to decide a vision for the kind of peer review you want in the context of your end-to-end publication workflow and then prioritise goals in order to reach this vision. A key part of this planning should be deciding how you will define and evaluate success. Have a clear framework for assessing success (‘of what on whom’, so on specific measures and specific population clusters). Systematically collect data and study the impact of the practice on journal performance. Key questions could be the following: is review quality improved? Is it more difficult to find reviewers? Are review times impacted? Are open reports being consulted and re-used? It is also advisable to consult with your journal community once the new process has been in place for some time, perhaps via survey, to gauge the development of their attitudes towards processes. Important here is to establish ex ante which quantifiable measures or performance indicators will be used for internal analysis. Outcomes should always be considered on an appropriate time scale, however. Change takes time.
F2. Accept that change takes time, but adjust if necessary
Bear in mind that cultural change takes time, and so, even where uptake is not as quick as wished, the broader ethical aims of transparency and accountability in scholarly publishing might make persistence desirable in spite of low uptake. However, if things really are not working, then it may be necessary to re-evaluate your goals in light of lessons learned. For example, revisit the advice in the ‘Set priorities and consider a phased approach’ and ‘Consider making options optional or piloting them first’ sections to consider phasing individual elements or making them optional.
F3. Share your results with the community
Giving updates on progress will enable community engagement, keeping authors, reviewers, editors and publishing staff updated on the progress of your initiative. These updates will also help others decide whether and how to implement similar approached. There is currently a lack of real scientific evidence on the efficacy of many traits of OPR. Once enough evidence has been gathered, consider writing up the results as a scientific study for peer-reviewed publication. Alternatively, consider partnering with peer review researchers from the start to ensure data is well-formed for such analyses and to enable rigorous external scientific analysis.