Selective citation in the literature on swimming in chlorinated water and childhood asthma: a network analysis

The number of citations that a publication receives is often regarded as an indicator of scientific quality [1]. The rationale behind this is that high-quality work will lead to more citations by peer scientists compared to low-quality work. It can be questioned, however, whether it is scientific merit alone that drives the number of citations or whether other factors have an influence as well.

Due to the large and growing number of scientific articles in the biomedical domain and limitation of the maximum number of references in many journals, it is not always feasible to cite all available literature. Ideally, a representative sample should be cited, but it is unclear on which ground researchers select the articles they cite. If this selection is not representative, but instead associated with specific characteristics of the cited literature, we speak of selective citation.

Citation bias is a special case of selective citation [2]. It concerns the selective citation of studies based on their outcome. Citation bias results in disproportionate attention for a specific segment of the literature that is in line with a hypothesis, while publications with an alternative view are systematically ignored. This can lead to a false consensus that is not evidence-based [3]. For instance, it has been shown that biased citation of previous evidence shapes the conclusions of reviews [4]. Citation bias can also lead to research waste by influencing funding decisions and guiding research in a wrong direction [4, 5].

Citation analyses can be performed in several manners. Greenberg [3] performed a network analysis based on a specific scientific claim: the relationship between a muscle disease called inclusion body myositis and beta-amyloid proteins. After identifying all the literature on this claim, he compared the number of citations to supportive empirical publications with the number of citations to critical ones. Greenberg concluded that 94% of the citations were assigned to the supportive data [3]. Next to a claim-specific network, a network analysis can also be based on a specific journal or time period. For instance, Fanelli [5] performed a citation analysis across several research domains. Articles were included if they tested any hypothesis and were published between 2000 and 2007. It was found that positive studies received on average 32% more citations than negative studies. However, this percentage differed substantially between the scientific domains [5].

In the current study, we performed a claim-specific citation analysis. We focused on the relation between swimming in chlorinated water and the development of childhood asthma.

Prior to performing this citation network analysis, we described the methodology in a study protocol and stored it at an online repository [13]. Deviations from this protocol are mentioned in the digital Additional file 1. In brief, we applied a search strategy to the Web of Science Core Collection, identified relevant literature, downloaded these records with their reference lists, extracted data for each article, built a dataset with potential citations, and used specialized software to determine which citations had occurred. These steps will be explained in more detail below.

First of all, if we look at the content-related article characteristics, there was some evidence for citation bias within this network: articles with a positive authors’ conclusion were cited more often than articles with a negative conclusion (odds ratio (OR) 1.8, 95% confidence interval (CI) 1.1–2.9). The data-based conclusion did not seem to impact the chance of citation. Articles with an empirical research design were cited more often than non-empirical ones (OR 4.2, CI 2.6–6.7), and a higher sample size was associated with a higher number of citations (OR 5.8, CI 2.9–11.6). Articles that specifically focused on our hypothesis under investigation, and thus fitted well in our network, were cited more often than those that did not (one step increase in specificity: OR 1.4, CI 1.2–1.7). However, the quality of the cross-sectional studies did not seem to have an impact on citation.

Other article characteristics that increased the number of citations were journal impact factor, number of authors, number of references, and reporting of a funding statement. Author characteristics that showed an impact on the number of citations were gender and region. The within-network authority of the cited article also increased the likelihood of citation (OR 4.1, CI 2.1–8.0), and there was strong evidence for self-citation in this network (OR 5.2, CI 3.1–8.8).

In addition, we tested whether articles with similar characteristics were more likely to cite each other. The results showed that concordance between articles did not have much impact on the likelihood of being cited, except possibly for concordance of study quality (Additional file 5).

We performed post hoc sensitivity analyses in which we excluded the narrative reviews and commentaries (with sample size being equal to 0) and adjusted for both study design and sample size (Additional file 6). The results were very similar to those of the complete data set. We performed an additional analysis in which we also excluded the ecological study and the case study (both with a wildly varying sample size). These results did show some stronger evidence for citation bias as studies with positive authors’ conclusions and data-based conclusions accumulate about three times more citations (Additional file 6). The results on the other determinants were fairly consistent with the previous findings.

The occurrence of citation bias was tested with two operationalizations of study outcome. The authors’ conclusion on the relation between swimming and asthma, as was often stated in the abstract, seemed to have an impact on citation, while the conclusion based on the underlying data seemed to have no impact. This implied that the authors’ conclusion and the data-based conclusion did not always concur. A post hoc chi square test showed that 40% of the negative results were interpreted as positive by the authors of those articles, while 0% of the positive results were interpreted as negative (data not shown; χ ² (1) = 6.3, p = 0.012).

Also, because self-citation seemed abundant in this field, we wondered whether particular researchers or research groups were responsible for this. First we assessed the number of research groups based on co-author relations of authors with at least three publications. Two research groups could be distinguished (Fig. 1). We then coded group membership for each article depending on which groups the authors belong to. Twelve articles stemmed from Bernard’s research group, nine articles came from the other identified research group, and 15 articles were written by authors that belonged to neither group. There were no articles in which the authors from both research groups collaborated. Interestingly, 11 of the 12 articles from Bernard’s research group had a positive authors’ conclusion, whereas the other research group only published articles with a negative or mixed/unclear conclusion (Additional file 7). Stratified logistic regression analyses showed that, indeed, self-citation occurred in both research groups (Additional file 7). This finding was confirmed by the self-citation rates of individual authors: the top 6 of self-citing authors stem from both of the research groups (Additional file 7).

Our research aim was to find out which determinants have an impact on the likelihood of being cited in the literature on swimming in chlorinated water and childhood asthma. We found that self-citation, large sample size, and empirical articles play a major role: in this network, they lead to a four to six times higher odds for citation. Other determinants associated with double citation odds or higher are journal impact factor, reporting of funding information, number of authors, authority, geographical region, and number of references.

There is some evidence for citation bias in this field of research. Articles in which the authors concluded that swimming in chlorinated water and asthma were related, are cited more often than those that did not. However, this finding is not robust and depends on the statistical test performed. Similarly, when we looked at the data-based conclusion rather than the authors’ conclusion, it did not have much impact on citation. This suggests that citation bias, if it occurs, is mostly based on interpretations and not so much on the underlying data. This would be in line with previous findings from our meta-analysis on citation bias, which showed that the authors’ conclusion has a higher impact on the likelihood of being cited than a data-based conclusion [2]. Still, the impact of study outcome on citation in the current network is small and not robust.

One explanation why we did not find strong evidence for citation bias might be due to the small size of the network. It is easy to have an overview of all published work in this field, without relying on the people that one knows, or on references that one happens to find. Also, this network seems to be quite balanced between proponents and opponents of the pool chlorine hypothesis, if we look at the number of positive [15] and negative articles [10]. This is contrary for instance, to the field of trans fatty acids and cholesterol, where 83% of all the published literature supports the view of an association, and citation bias was found to be much more prevalent in this field (Urlings et al.: Citation bias in the trans fatty acid literature: evidence from a citation network analysis on the effect of industrially produced trans fat and cholesterol, submitted). Such balance in viewpoints secures the possibility of adequate contradiction so that no point of view disappears until the debate is settled.

This debate, on whether swimming in chlorinated water increases the chance of developing asthma later during childhood, could best be settled by means of longitudinal research. Nevertheless, the majority of empirical articles used a cross-sectional design, and one third of all articles on this topic is not even empirical. We identified only three prospective cohort studies within the network [35, 36, 46], one of which did not assess asthma per se but lung function in general. Of the remaining two studies, Voison et al. concluded that they had found support for the positive association between swimming and asthma development [46], whereas Font-Ribera et al. concluded the opposite that there was a smaller chance of current asthma for children who had swum more [35].

Our citation analysis has several limitations. First of all, the network is small relative to the number of predictors, which makes it vulnerable for chance findings. Similarly, some research design categories were represented by only one publication. Secondly, there was a high degree of multicollinearity between study design and sample size, which made it impossible to adjust for all the factors we had originally planned. We dealt with these two limitations by running several sensitivity analyses, with different adjustments and with inclusion of different research designs. Thirdly, asthma is difficult to diagnose, and in the literature, we found different ways to assess asthma, with a varying degree of validity. To make these articles compatible, we combined these different assessments into one measure for data-based conclusion on asthma.

Our network consists of publications identified by the Web of Science Core Collection. This database offers the option to download reference lists as part of the search output. Other databases, such as MEDLINE and Embase, do not provide this option yet. Because these reference lists are needed by the software we used (CitNetExplorer) to build a citation network, we did not include these other databases in our search strategy. It is therefore likely that we missed parts of the literature. However, we see no reason why the citation dynamics in the Web of Science Core Collection would be different from other databases. Nevertheless, it would be an improvement if reference lists were added to the search output of these other databases so that this literature can be included in future citation networks.

There is clear evidence of selective citation in the research field studying the relation between swimming in chlorinated water and childhood asthma: several factors have an impact on the chance of being cited. Authors particularly prefer to cite their own work over those of other authors on the same topic. The evidence for citation bias in this field was not very strong and inconsistent. The impact of citation bias on the development of knowledge therefore seems limited in this field.