Spin in the reporting, interpretation, and extrapolation of adverse effects of orthodontic interventions: protocol for a cross-sectional study of systematic reviews

Abstract

Background

Titles and abstracts are the most read sections of biomedical papers. It is therefore important that abstracts transparently report both the beneficial and adverse effects of health care interventions and do not mislead the reader. Misleading reporting, interpretation, or extrapolation of study results is called “spin”. In this study, we will assess whether adverse effects of orthodontic interventions were reported or considered in the abstracts of both Cochrane and non-Cochrane reviews and whether spin was identified and what type of spin.

Methods

Eligibility criteria were defined for the type of study designs, participants, interventions, outcomes, and settings. We will include systematic reviews of clinical orthodontic interventions published in the five leading orthodontic journals and in the Cochrane Database. Empty reviews will be excluded. We will manually search eligible reviews published between 1 August 2009 and 31 July 2019. Data collection forms were developed a priori. All study selection and data extraction procedures will be conducted by two reviewers independently. Our main outcomes will be the prevalence of reported or considered adverse effects of orthodontic interventions in the abstract of systematic reviews and the prevalence of “spin” related to these adverse effects. We will also record the prevalence of three subtypes of spin, i.e., misleading reporting, misleading interpretation, and misleading extrapolation-related spin. All statistics will be calculated for the following groups: (1) all journals individually, (2) all journals together, and (3) the five leading orthodontic journals and the Cochrane Database of Systematic Reviews separately. Generalized linear models will be developed to compare the various groups.

Discussion

We expect that our results will raise the awareness of the importance of reporting and considering of adverse effects and the presence of the phenomenon of spin related to these effects in abstracts of systematic reviews of orthodontic interventions. This is important, because an incomplete and inadequate reporting, interpretation, or extrapolation of findings on adverse effects in abstracts of systematic reviews can mislead readers and could lead to inadequate clinical practice. Our findings could result in policy implications for making judgments about the acceptance for publication of systematic reviews of orthodontic interventions.

Background

Readers of the biomedical literature mostly just screen the title and the abstract of an article without assessing the full publication [1]. The beneficial and adverse effects of interventions should therefore be transparently reported in these summaries and should not mislead its readers. Misleading reporting, interpretation, or extrapolation of study results is called “spin” [2,3,4]. We will assess in abstracts of both Cochrane and non-Cochrane reviews whether adverse effects of orthodontic interventions were reported or considered and whether spin was identified and what type of spin.

Titles and abstracts are the most read sections of biomedical papers [1], because assessing the full research article is often conditioned by paywalls or because of a lack of time or language issues of the readers [1]. Abstracts should therefore clearly and truthfully reflect the objectives, methods, results, and the interpretation of research findings. The standard for Methodological Expectations of Cochrane Intervention Reviews (MECIR) [5] has listed a series of highly desirable and mandatory items that should be consulted by reviewers when preparing the abstract of their reviews. Item R13 of the MECIR standard states that: “The Abstract of the review should aim to reflect a balanced summary of the benefits and harms of the intervention.” This mandatory item is particularly crucial for presenting adverse effects of health care interventions, because these effects are often poorly reported in systematic reviews [6]. Numerous epidemiological studies have also shown that the assessment and reporting of adverse effects of interventions in primary research studies is often suboptimal [7,8,9,10,11]. We adopted Cochrane’s definition of adverse effects: “An adverse event for which the causal relation between the intervention and the event is at least a reasonable possibility” [12, 13]. This definition and other key terminology in this manuscript are summarized in Table 1 [2,3,4, 12,13,14,15].

Table 1 Glossary of terms

When presenting information on adverse effects in the abstract, it is also crucial that it does not mislead the reader. A distorted presentation of study results has been defined as “spin” [3], but more elaborate definitions are also used (Table 1). The term spin was first used in 1995 in the medical literature by Horton [16] and has been further subdivided into three categories [4]: misleading reporting-related spin, misleading interpretation-related spin, and misleading extrapolation-related spin (Table 1). Yavchitz et al. [17] have ranked the various types of spin according to their severity. The severest form of spin in abstracts of systematic reviews and meta-analyses was scored for “conclusions that contain recommendations for clinical practices that were not supported by findings” [17]. A high prevalence of the various types of spin has been identified in multiple epidemiological studies [4, 18,19,20,21,22]. Boutron et al. [18] found spin in 50% (36/72) of the conclusions sections of the main text of parallel-group RCTs and in 58.3% (42/72) of the conclusions sections of the abstracts. Spin was also common in diagnostic accuracy studies published in journals with high impact factors [22]. Lockyer et al. [21] showed that spin is a frequent phenomenon in abstracts of RCTs of wound treatments, and Lazarus et al. [4] identified at least one example of spin in 84% (107/128) of the abstracts of non-randomized intervention studies. Spin is in strong conflict with the Declaration of Helsinki [23] that states that: “Authors have a duty to make publicly available the results of their research on human subjects and are accountable for the completeness and accuracy of their reports.”

In this study, we will assess whether potential adverse effects of orthodontic interventions were reported or considered (i.e., discussed, weighed, etc.) in the abstract of systematic reviews. We will further assess whether spin was introduced regarding information on these adverse effects in the abstract, and we will categorize the types of spin (Table 1). We will assess these issues in the five leading orthodontic journals and those included in the Cochrane Database of Systematic Reviews. In these reviews, we will assess adverse effects such as pain as a result of tooth movement and the adverse effects defined by Preoteasa et al. (Table 2) [24]. Scoping searches in the orthodontic literature confirmed the knowledge gaps on our research questions. Our pilot studies quantified these gaps and confirmed the need to address these questions. We will assess these issues in systematic reviews, because they are increasingly consulted by patients [25] and when well-conducted systematic reviews are considered among the information sources with the highest level of evidence [26]. Our research questions are important, because incomplete or misleading information on adverse effects of interventions may have detrimental effects on the treatment of orthodontic patients.

Table 2 Adverse effects hypothetically linked to orthodontic interventions [24]

Objectives

The objectives of this research study are summarized in the following research questions:

Research questions

• In abstracts of systematic reviews of orthodontic interventions, were potential adverse effects of these interventions reported or considered (i.e., discussed, weighed, etc.)?

• In abstracts of systematic reviews of orthodontic interventions, was spin identified in the reporting, interpretation, and extrapolation of adverse effects?

• What was the prevalence of each type of spin?

Methods

This protocol is reported according to the guidance of the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 statement, and the PRISMA-P checklist is included as Additional file 1 [27, 28]. We adopted the same flow of research methods as reported in our published protocol on seeking adverse effects in systematic reviews of orthodontic interventions (Fig. 1) and conducted our pilot tests on the same sample of systematic reviews as was described in our previous protocol [15]. Our sample size of 14 reviews for the pilot test was calculated with the formula reported by Viechtbauer et al. [29]. Further details on the methods of our pilot test are reported in Additional file 2. This pilot test found that the reviewers in only 35.7% (5/14) of the abstracts reported or considered (i.e., discussed, weighed) potential adverse effects of orthodontic interventions. This sample identified an overall prevalence of 14.3% (2/14) of spin in the abstract on adverse effects of orthodontic interventions. Both cases of spin were “misleading reporting-related spin.” The following sections describe our planned methods based on these pilot tests. We will not start the selection of eligible reviews and data extraction prior to the complete acceptance of this protocol for publication.

Fig. 1

Flow diagram of the research methods. “Flow diagram of the research methods” was published previously by Steegmans et al. [15] in the journal “Systematic Reviews,” which is an open access journal of BioMed Central. Copyright on any open access article in a journal published by BioMed Central is retained by the author(s)

Eligibility criteria

We will adopt the same eligibility criteria that were defined for our published protocol on seeking adverse effects in systematic reviews of orthodontic interventions [15]. To avoid misinterpretation, we copied and pasted these eligibility criteria into Table 3 [15, 30].

Table 3 Eligibility criteria

Information sources

We will manually search the Cochrane library [14] and the websites of the five leading orthodontic journals to identify eligible systematic reviews published between 1 August 2009 and 31 July 2019. We chose this starting date because the first of August 2009 coincides with the launch of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement and its guidance paper on 21 July 2009 [27, 28]. Journal selection was based on two criteria: (1) the journal has been published for 10 years or more and (2) the impact factor. The journal citation reports by Clarivate Analytics were consulted to identify the five leading orthodontic journals based on impact factor [31]. The following five orthodontic journals fulfilled both criteria: European Journal of Orthodontics (EJO), American Journal of Orthodontics and Dentofacial Orthopedics (AJODO), Angle Orthodontist, The Korean Journal of Orthodontics, and Orthodontics and Craniofacial Research.

Study records

Data management

• Two authors (PS and RMR) will conduct all study selection and data extraction procedures independently.

• Pilot tests were conducted to train both reviewers in applying these methods consistently and for calibration purposes [28].

• We will apply the following strategies in the case of disagreement between the two authors on the eligibility of a paper or the extraction of data: (1) discussions between reviewers, (2) rereading the paper, (3) or if necessary contacting its authors [32]. Persistent disagreements will be resolved through the consultation and arbitration of a methodologist (NDG).

• All eligible systematic reviews will be downloaded, and all extracted data will be collected in an Excel spreadsheet.

Selection process

• All titles and abstracts in the websites of the five orthodontic journals will be hand-searched to identify eligible reviews. The section “Dentistry and Oral health” will be searched in the Cochrane library for eligible Cochrane reviews [14].

• We will only include the latest version of a review when updates have been published.

• Authors will be contacted in the case of doubt regarding multiple publications of the same review. We plan to include the first publication, but will make this decision on a case by case basis and will report the rationale for this choice.

• Our selection procedures will be presented in a PRISMA flow diagram [32, 33].

• All included and excluded studies will be presented in tables, and the rationale for exclusion will be given for each excluded review.

Data collection process

• All eligible studies together with their supplemental files will be merged into binder PDFs, and pertinent search terms are linked to these documents to facilitate data extraction [34, 35].

• Eligible search terms were identified through searches in thesauri and in key articles on adverse effects [13, 36,37,38,39,40]. These terms are given in Additional file 3 and are identical to those used in our protocol on seeking adverse effects in systematic reviews of orthodontic interventions [15].

• Our pilot-tested data collection forms will be used for all data extraction procedures (Additional file 4). The PRISMA [32, 33] and the PRISMA-P [27, 28] checklists and guidance were consulted to develop these forms. The criteria for scoring pertinent data items are defined in these forms.

• The entire eligible review of both orthodontic and Cochrane reviews will be searched for data items, i.e., the text, tables, figures, and all supplemental files. The plain language summary in the eligible Cochrane reviews will not be searched for data items.

• When during the data extraction procedure changes are made in the data collection forms, we will present this with rationale in the section “Differences between the protocol and review.”

Scoring adverse effects of orthodontic interventions

• We will use the framework of categories of known orthodontic adverse effects as defined by Preoteasa et al. [24] (Table 2). We will also include pain as a result of tooth movement as an adverse effect. Potential adverse effects that are identified during data extraction will be discussed between the two reviewers (PS and RMR). We will report the rationale when including additional adverse effects and will categorize them.

• Ambiguous outcomes that could be interpreted as either a beneficial or an adverse outcome will not be scored as “adverse.” The rationale for this score will be given. We will only consider ambiguous outcomes as “adverse” when the review authors define these outcomes as such and make a strong case for this classification.

Scoring spin in the reporting, interpretation, and extrapolation of adverse effects of orthodontic interventions

• We will assess three types of spin, i.e., misleading reporting, misleading interpretation, and misleading extrapolation on adverse effects of orthodontic interventions in the abstract (Table 4). Each type of spin will be assessed separately for reviews that either did or did not seek adverse effects of interventions.

• To facilitate our scoring procedures and to reduce the risk of misinterpretation, we subdivided each type of spin into categories and defined each category (Table 4). We will score the presence of spin when spin is identified for one or more of these categories. The scoring procedures are summarized in Additional file 4. Pilot tests were conducted to assess the validity of these procedures.

Table 4 Types of spin in reviews that did or did not seek adverse effects of interventions

Outcomes and statistical analyses

• Figure 2 a and b present all research questions in a flow diagram, and Table 5 lists all planned outcomes.

• We will calculate and report all prevalence data with their 95% confidence levels.

• We calculate the prevalence statistics for (1) all journals as one group, (2) the group of five leading orthodontic journals and the Cochrane Database of Systematic Reviews separately, and (3) each individual journal separately. Generalized linear models will be developed having the following outcomes for the abstracts of systematic reviews of orthodontic interventions: the reporting or considering of potential adverse effects of interventions/no reporting or considering of potential adverse effects of interventions (binary); presence of SPIN/absence of “SPIN” (binary); and misleading reporting/misleading interpretation/misleading extrapolation/no SPIN (categorical). The models will account for journal category (Cochrane Database of Systematic Reviews vs others), individual journals, and the geographical location of the study. Statistical significance will be based on a p value < 0.05. Stata software (Stata Corporation, College Station, TX, USA) version 15 will be used for all the statistical analyses [41].

• All outcomes that will be introduced or eliminated post hoc will be reported together with the rationale for inclusion or exclusion.

Fig. 2

a Reporting or considering adverse effects of orthodontic interventions in the abstract. b Spin on adverse effects of orthodontics in the abstract

Table 5 Summary of findings

Reporting of the research study and data management

• The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement will be used as the guideline for reporting the completed cross-sectional study [42].

• A data management plan was prepared for the long-term storage of our research data [43] in the case that the publisher of our completed research study will not or will only partly store our raw data. We consulted the Registry of Research Data Repositories [44] to identify an appropriate repository for our type of research data. We selected Dryad [45] for two reasons: (1) it is an international repository of data of peer-reviewed scientific and medical research and (2) it also includes data sets for which no specific data repository exist such as meta-epidemiological research data of systematic reviews in orthodontics. Our data management plan implies that (1) all our research data will be made freely available, (2) our completed article will present a link to a repository in which all raw data of the study will be deposited, (3) the repository is registered in the Registry of Research Data Repositories [44], (4) our research data will be reported in a format that permits other researchers to understand, cite, and reuse these data, (5) all sensitive data will be protected, and (6) it will be reassessed frequently and also updated if necessary [43, 44].

Differences between the protocol and the completed study

• We will report all modifications between the protocol and the final research study. The rationale for each of these changes will be given.

• We will also report the consequences of these modifications on the magnitude, direction, and the validity of the outcomes [46].

Discussion

Strengths

Key strengths of this research study include the following: (1) we conducted extensive scoping searches and pilot studies to fine-tune our research questions and methods. These activities confirmed the importance of our questions. (2) Our research team consists of two topic experts (PS and RMR) and two methodologists (RMR and NDG). (3) All study selection and data collection procedures will be undertaken independently by two authors (PS and RMR). Calibration of these operators was done during the pilot studies. (4) To guarantee reproducibility and full access to our data, we will publish our protocol a priori and will include all raw data of the completed research study in additional files or will deposit them in an open-access repository [43,44,45, 47].

Limitations

Including only orthodontic intervention reviews published in the five leading orthodontic journals and in the Cochrane Database of Systematic Reviews could be a limitation, but we expect that the findings in this subgroup of journals will underestimate the true severity of spin on adverse effects of interventions in the abstracts of these reviews. Including only reviews published in the last 10 years could also be a limitation. However, we chose this period because it brings the current knowledge status on our research questions to the foreground and these 10 years coincide with the launch in 2009 of the checklist of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [32, 33].

Importance and beneficiaries

In this research study, we will address three key questions in abstracts of systematic reviews of orthodontic interventions: whether potential adverse effects of these interventions were reported or considered, whether spin was identified regarding information on these adverse effects, and the type of spin. These issues are important, because (1) the assessment and reporting of adverse effects of interventions is often suboptimal [7,8,9,10,11], (2) titles and abstracts are the most read sections of papers in the biomedical literature [1], (3) a high prevalence of spin has been identified in abstracts of both randomized and non-randomized studies [4, 21], and (4) incomplete or inadequate reporting, interpretation, or extrapolation of findings on adverse effects in the abstract can mislead readers and could lead to inadequate practice [4]. Our results will raise the awareness of considering adverse effects and the phenomenon of spin regarding these effects in abstracts of systematic reviews of orthodontic interventions. Patients, clinicians, researchers, editors, peer-reviewers, guideline developers, policy makers, and research funders will all be beneficiaries of the findings of this research study.