We have adopted the general method proposed by the EQUATOR Network for developing reporting guidelines [11]. The process for ACCORD development is outlined in Fig. 2.
Stage 1: establishment of a Steering Committee
With the endorsement of the International Society of Medical Publication Professionals (ISMPP), we assembled a Steering Committee to develop a reporting guideline for research using consensus. The Steering Committee (the authors, AH, AP, CW, DT, EH, EvZ, KG, NH, PL, RM, and WG) will lead and co-ordinate the guideline development process. Specifically, the Steering Committee will be responsible for: establishing the goals and timelines for the work, including registering and publishing the protocol; generating the initial list of checklist items from the literature review; conducting a consensus process to enrich and refine the initial list of minimum items that should be reported; implementing each stage of the process including developing questionnaires and analysing voting outcomes and other data; reporting the findings of the process in a statement document with the main checklist and guidance; developing an E&E document where all the items are individually explained and examples of approach and reporting are given; disseminating the reporting guidelines via publication, presentation at congresses and other events, and online presence including a website linked to social media channels.
The Steering Committee is a multidisciplinary group (11 people) that includes clinician practitioners, methodologists, publication professionals, patients, journal editors and publishers and the pharmaceutical industry. Prior to initiating Stage 2, we listed the project in the EQUATOR Network registry for reporting guidelines under development [50] and registered the protocol with the Open Science Framework [51].
Stage 2: literature review and generation of draft checklist items
The aim of this step is to seek evidence on the quality of reporting of the process undertaken in health studies using consensus methodology. This research will provide insight into possible checklist items for evaluation by the Delphi Panel (further information on the Delphi Panel is provided in ‘Stage 3’ below). The CREDES guidelines, specific to palliative care, will also be reviewed for elements that can be generalised to other biomedical fields [38].
Search strategy
The process for conducting the systematic review will be informed by and reported according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) 2020 and PRISMA-Search extension guidelines [11, 52]. Eligible studies will include studies, reviews and published guidance addressing the quality of reporting of consensus methodology that aim to improve health outcomes in biomedicine or clinical practice. Reports of studies using consensus methods but not commenting on their reporting quality will be excluded, for example, studies to reach clinical recommendations of core outcome sets or reporting guidelines using consensus methods. Ineligible publications include editorials, letters about individual publications, and comments on methodology of consensus outside the scope of biomedical research.
Searches of EMBASE (OVID), MEDLINE (OVID), Web of Science - Core Collection, MEDLINE (Web of Science), PubMed, Cochrane Library, and Emcare (OVID), Academic Search Premier and PsycINFO databases will be run with no limits by year or language of publication at the search stage. Four initial search strategies were developed and sequentially piloted by members of the Steering Committee (WG, EvZ and PL) with the assistance of an information (JS) and systematic review specialist (ZF). The piloting allowed the adjustment of the initial search strategy by the information specialist to provide results that better aligned with the inclusion criteria and objective of this study. The refined, broad search strategy (Supplementary File) will be used to identify and generate the final list of studies focusing on the quality and accuracy of reporting of Delphi and other consensus processes, methods, techniques or recommendations. The search may also be augmented with relevant articles highlighted by the Steering Committee as appropriate based on the individuals’ prior work and expertise in the area (via a manual search).
Data extraction
EvZ, PL, WG, and ZF will independently screen the titles and abstracts retrieved from the search for potential inclusion using the Rayyan tool in blind mode [53]. Any discrepancies will be resolved by discussion. Full-text articles will then be retrieved and assessed independently for eligibility, with reconciliation of any differences through discussion. Data will be extracted using a draft extraction form, which will be piloted on three studies before use. Based on the information gathered on the literature review, a list of preliminary items for the checklist will be generated to be refined in a Delphi exercise in Stage 3.
Stage 3: reaching consensus on checklist items
We will use Delphi methodology, as described below, to reach a consensus regarding the checklist items to include in the reporting guideline. This will take place in two steps, with the first involving the Steering Committee and the second involving a full Delphi Panel (the ACCORD Delphi Panel; Fig. 3). We plan to report the consensus methodology in accordance with our own guidelines under development.
First step: steering committee survey
The Steering Committee will review the data extracted from literature search. This initial list is likely to contain duplicated items or items that require rewording. The aim is to eliminate repetitions and inadequately or ambiguously written items to reach a list of unique items. Using a survey, the Steering Committee members involved in the literature review will independently suggest items for the initial checklist; NH and WG will consolidate the initial checklist items.
There will then be anonymous voting to confirm the initial checklist that will be put to the full ACCORD consensus panel. Steering Committee members (excluding NH and WG) will vote (anonymised and blinded) on whether they ‘Strongly Agree’, ‘Agree’, ‘Disagree’, ‘Strongly Disagree’, or feel ‘Abstained/Unable to answer’ for all proposed items. There will also be the opportunity to provide comments. Any items that do not receive support will be discussed by the Steering Committee, and either included as ‘possible additional items’ or discarded completely. The eliminated items and the reasons for their elimination will be reported. The candidate items will be presented in sequence as a draft checklist, and in the same order to all people voting, so that the overall checklist structure, considering the manuscript sections (such as Introduction, Methods, Results, Discussion) can be evaluated. Within each section, there will be ‘proposed items’ and ‘possible additional items’.
Second step: ACCORD Delphi panel
The preliminary list of checklist items agreed on by the Steering Committee will subsequently be put to the ACCORD Delphi Panel for validation using a blinded electronic voting platform (e-survey). In addition, the ACCORD Delphi Panel will be provided with the list of items excluded by the Steering Committee for information, as a confirmatory step.
The order of the candidate items within each manuscript section will be randomised so that it is different for each person voting and all items are evaluated fully independently from each other. Five voting options will be offered: ‘Strongly Agree’, ‘Agree’, ‘Disagree’, ‘Strongly Disagree’, and ‘Abstained/Unable to answer’. Votes of ‘Abstained/Unable to answer’ will be included in the denominator. Panellists will be able to provide free-text comments and will have the opportunity to propose additional items. There will be three rounds of voting; with feedback and descriptive statistics incorporated for the next round by NH and WG. The approval rate and the reasons for elimination of items will be reported.
The consensus threshold is defined in this step as at least 20 respondents (approximately 50% of the target panel size), and at least 80% of responding ACCORD Delphi panellists who are able to answer voting ‘Agree’ or ‘Strongly Agree’, with two rounds of statement revision and re-voting. The Steering Committee will review items that do not achieve consensus in rounds 1 or 2 and these will be revised or eliminated taking into account any free-text comments. If consensus is not achieved by the ACCORD Delphi Panel, or there are insufficient respondents, the Steering Committee may decide that the item will be included as an optional item or a discussion point on the E&E document or checklist, alongside core items on which consensus was achieved. Simple descriptive statistics (response rates, level of agreement for each statement, median levels of agreement and interquartile ranges) will be used to describe approval rates between rounds. The same measures will be used to evaluate consensus stability across rounds [54].
There are no generally agreed standards for the panel size for Delphi studies, and a wide range of panel sizes has been reported; panels of 20–30 participants are common [55, 56]. However, it is recognised that the size and diversity of a Delphi panel can impact the quality of the final recommendations [57]. The ACCORD Delphi Panel will comprise approximately 40 members, so that it allows for representation from clinicians, methodologists, patient advocates, lay public representatives, health technologists, journal editors and publishers, regulatory specialists, and publications professionals, and to ensure an acceptable number of responses (20, or at least 50% of the group) in the event of drop-outs or partial completion of review. The ACCORD project will be advertised to potential Delphi Panellists via relevant societies, organisations, and networks; in addition, authors of recently published consensus studies in high-profile journals will be invited directly.
When registering, panellists will be asked to complete a preliminary survey to capture basic information on experience, geographical, and demographic representation. Although no formal targets will be established, the Steering Committee will endeavour to ensure a broad spread of representation across these categories. Members of the Delphi Panel will be recognised as contributors in the acknowledgements section of the guideline (with their permission) but participation in ACCORD Delphi panel will not qualify a panellist for authorship.
Software or a voting platform that is appropriate for Delphi exercises will be used to implement the voting process, administered by NH and WG. Alternatives available on the market are being evaluated and tested at the time of this protocol publication, and the platform and version used will be reported. Initial requirements are that the software used follows security regulations, ethical standards and allows, besides voting, the inclusion of free text responses in the e-surveys to supplement discussion in the E&E document.
Stage 4: creation of the reporting guideline and E&E document
On completion of the Delphi consensus process, the checklist will be finalised by WG and NH for approval by the Steering Committee, and the reporting guideline will be developed. A separate E&E document will be created to provide a detailed rationale for the items included in the checklist. In each case, an example will be included of good reporting from a published paper. The E&E document can also be informed by perspectives collected from researchers involved in consensus-based studies outside the biomedical field.
Stage 5: dissemination
We intend to publish the reporting guideline and E&E document in open access format via a CC-BY copyright licence. Future publications from the ACCORD project will be reported according to the best available reporting guidelines for each type of manuscript. To aid dissemination, we plan to present the findings at congresses including ISMPP European and Annual Meetings, the World Conference on Research Integrity and Peer Review, and the UK Research Integrity Office Annual Conference. Progress will be updated on a dedicated website for the ACCORD project, the EQUATOR website and newsletter, and social media channels, and communicated in appropriate professional forums and events. This dissemination of the reporting guideline is crucial for the document to be implemented in practice.