The purpose of this study was to assess the degree to which paper and poster presenters at the 6th World Conference on Research Integrity ([17]; http://wcri2019.org/) preregistered their research on research integrity using the six key elements described the Amsterdam Agenda (AA; https://www.wcrif.org/guidance/amsterdam-agenda). The AA was developed and adopted at the 5th WCRI ([16]; https://wcrif.org/wcri2017) and recommended for use in research on research integrity. NS and LB were involved in drafting and finalizing the AA. KS analysed the compliance with the AA of the authors of papers and posters accepted for the 6th WCRI (www.wcrif.org/images/2019/PDF/Abstract_book.pdf) and presented his findings during the final plenary session of the conference.
Preregistration can vary from a minimum set of details on the study at issue to a full study protocol, with or without a complete data-analysis plan. For example, Wagenmakers et al. [13] defined preregistration as posting a study protocol including research questions, research design, statistical methods and data-analysis plan in a repository prior to collecting and analyzing the data [13]. WCRI participants could use a number of repositories for preregistering research on research integrity, including: Open Science Framework (OSF) (https://osf.io), Dataverse (https://dataverse.org/), Figshare (https://figshare.com/), and Mendeley (https://www.mendeley.com/). The AA provided guidance on how to register a study with OSF but left the choice of the repository open. We note that the Amsterdam Agenda not only called for preregistering research on research integrity, but also suggested to do so in the Registry for Research on the Responsible Conduct of Research. No one followed that second recommendation and in hindsight, it seems not to be a good idea to have a separate registry for one specific type of study. Furthermore, it is not so clear which studies would belong in such a registry. What is considered as research on research integrity is somewhat a moving target, although probably many would agree that its description by the WCRF Foundation (www.wcrif.org/foundation/mission) accurately characterizes the current situation.
Repositories time-stamp the original document and it later updates so that it can be seen whether potentially data-driven changes of research questions, research design, statistical methods and data-analysis plan were made. The access to the preregistered study protocol can be made public or access can be restricted or embargoed. Based on the relevant literature, our own conceptual analysis, and the interaction we had with the participants before and during the 5th WCRI, we identified the following five core objectives of preregistration.
-
(1)
Limiting Researcher Degrees of Freedom. Preregistration limits undesirable flexibility in the analysis and interpretation of the results and thus prevents selective reporting [12]. Several authors have convincingly argued that data analysis in the health sciences as well as in the social and behavioral sciences, economics and in other areas easily elicits activities from researchers known as “p-hacking” [14], “fishing” [13], “cherry picking” [5], and “HARKing” (i.e., Hypothesizing After the Results are Known [8];). All these phenomena have in common that they can help to obtain spurious positive results by utilizing undesirable flexibility or ‘researcher degrees of freedom’. Ioannidis [7] convincingly explained why deriving research results from analyzing data without a prespecified plan produces very often results that fail to stand firm upon replication (also, [9, 11]).
-
(2)
Ensuring Replicability. The detailed study protocol that preregistration requires, ensures that others can repeat the study based on the information the researcher preregistered [10].
-
(3)
Revealing Unpublished Studies. Preregistration that is publicly accessible allows authors of systematic reviews to assess the magnitude of publication bias due to (often negative) studies of which the results weren’t published [4].
-
(4)
An improved version of preregistration are registered reports (RR [15];). In RR the introduction and method sections are submitted to a journal before data collection starts. Reviewers assess the study plan, including the research questions, research design, statistical methods and data-analysis steps envisaged, and the journal editor makes a decision about acceptance of the research for publication solely on the basis of the relevance of the study and the soundness of its methods. This version of preregistration highlights a fourth core idea:
-
(5)
Greater Review Benefit. Reviewers’ comments on RR submissions may improve the study because its design can still be adapted [3].
-
(6)
Protecting Reviewers and Editors from Bias. RRs also prevent the journal editor and the reviewers from letting the research outcomes, unknown at the time of review and decision making, affect their decision of whether to publish the study, and thus prevents publication bias and overrepresentation of positive results. This benefit is supported by the fact that final publications preceded by RRs mention negative results substantially more often than comparable articles that were not preregistered [1]. Nosek et al. [10] support preregistering research; that is, committing oneself to an a priori study protocol. They conclude that without preregistration statistical analyses are rather meaningless and could better be avoided.
The World Conferences on Research Integrity foster the exchange of information and discussion about responsible conduct of research (https://wcrif.org/). Beginning with the 2010 Singapore Statement, the World Conferences have adopted conference statements that are designed to foster or improve integrity in research. The Amsterdam Agenda was developed during the 5th conference in 2017, with the goal of establishing a Registry for Research on the Responsible Conduct of Research (RRRCR; https://osf.io/jbqkv/; part of OSF). The registry was focused especially on researchers engaged in studying research integrity in all its facets and aimed explicitly at having researchers use the six key elements outlined in the AA when developing their study protocol. These key elements were made available to the participants of the 6th WCRI and can be found on the organization’s website (https://www.wcrif.org/guidance/amsterdam-agenda); they are:
-
Problem. The integrity researcher should describe the shortcomings one addresses, such as selective reporting or the misuse of statistics.
-
Impact. One should estimate the impact the target shortcomings have on the trustworthiness of research, how they affect the responsible use of research funds, etc.
-
Intervention. Researchers should explain how they plan to address shortcomings when identified. Examples of actions are future quality checks, training of researchers to prevent future shortcomings, and encouragement of responsible behavior.
-
Hypothesis or Anticipated Outcomes. Researchers should explain the changes in future research they expect their intervention will cause.
-
Assessment. One should explain the planning of hypothesis testing and assessing whether outcomes are as expected.
-
Data sharing. Finally, the registry requires researchers to clarify how data, either qualitative or quantitative, will be shared with other researchers.
To preregister a study, researchers were expected to provide information on each of the six key elements, upload a full study protocol, and, once the study was completed, upload a data set from the study and reports describing results. Preregistration of studies on which an abstract was submitted for the 6th WCRI was voluntary.