Publishing computational research - a review of infrastructures for reproducible and transparent scholarly communication

Background The trend toward open science increases the pressure on authors to provide access to the source code and data they used to compute the results reported in their scientific papers. Since sharing materials reproducibly is challenging, several projects have developed solutions to support the release of executable analyses alongside articles. Methods We reviewed 11 applications that can assist researchers in adhering to reproducibility principles. The applications were found through a literature search and interactions with the reproducible research community. An application was included in our analysis if it (i) was actively maintained at the time the data for this paper was collected, (ii) supports the publication of executable code and data, (iii) is connected to the scholarly publication process. By investigating the software documentation and published articles, we compared the applications across 19 criteria, such as deployment options and features that support authors in creating and readers in studying executable papers. Results From the 11 applications, eight allow publishers to self-host the system for free, whereas three provide paid services. Authors can submit an executable analysis using Jupyter Notebooks or R Markdown documents (10 applications support these formats). All approaches provide features to assist readers in studying the materials, e.g., one-click reproducible results or tools for manipulating the analysis parameters. Six applications allow for modifying materials after publication. Conclusions The applications support authors to publish reproducible research predominantly with literate programming. Concerning readers, most applications provide user interfaces to inspect and manipulate the computational analysis. The next step is to investigate the gaps identified in this review, such as the costs publishers have to expect when hosting an application, the consideration of sensitive data, and impacts on the review process.


Introduction
Many scientific articles report on results based on computations, e.g., a statistical analysis implemented in R. Publishing the used source code and data to adhere to open reproducible research (ORR) principles (i.e., public access to code and data underlying the reported results (Stodden et al. 2016)) seems simple.However, several studies concluded that papers rarely link to these materials (Stagge et al. 2019;Nüst, Granell, et al. 2018).Moreover, due to technical challenges, e.g., capturing the original computational environment of the analyst, even accessible materials do not guarantee reproducibility (Chen et al. 2018;Konkol, Kray, and Pfeiffer 2018).These issues have several implications (Morin et al. 2012): It is difficult (often even impossible) to find errors within the analysis, but publishing erroneous papers can damage an author's reputation (Herndon, Ash, and Pollin 2013) as well as trust in science (National Academies of Sciences, Medicine, and others 2019).Also, reviewers cannot verify the results, because they need to understand the analysis just by reading the text (Bailey, Borwein, and Stodden 2016).Furthermore, other researchers cannot build upon existing work but have to collect data and implement the analysis from scratch (Powers and Hampton 2018).Finally, libraries cannot preserve the materials for future use or education.These issues are also to society's disadvantage as it cannot benefit fully from publicly funded research (Piwowar and Piwowar 2007).Fortunately, funding bodies, e.g., Horizon 2020 (https://ec.europa.eu/research/participants/docs/h2020-funding-guide/cross-cutting-issues/open-access-dissemination_en.htm, last access for this and the following URLs: 20th Dec 19), increasingly consider data and software management plans as part of grant proposals.Accordingly, more editors add a section on code and data availability into their author guidelines (see, e.g., Nüst, Ostermann, et al. (2019); Hrynaszkiewicz (2019)), and reviewers consider reproducibility in their decision process (Stark 2018).Nevertheless, these cultural and systematic developments (Munafò et al. 2017) alone do not solve the plethora of reproducibility issues.Authors often do not know how to fulfill the requirements of funding bodies and journals, such as the TOP guidelines (Nosek et al. 2015).It is important to consider that the range of researchers' programming expertise varies from trained research software engineers to self-taught beginners.For these reasons, more and more projects work on solutions to support the publication of executable supplements.The key contribution of this paper is a review of applications that support the publication of executable computational research for transparent and reproducible research.This review can be used as decision support by publishers who want to comply with reproducibility principles, editors and programme committees planning to adopt reproducibility requirements in the author guidelines and integrate code evaluation in their review process (Eglen and Nüst 2019), applicants in the process of creating data and software management plans for their funding proposals, and authors searching for tools to disseminate their work in a convincing, sustainable, and effective manner.We also consider aspects related to preservation relevant for librarians dealing with long-term accessibility of research materials.Based on the survey, we critically discuss trends and limitations in the area of reproducible research infrastructures.
Scope: This work focuses on applications that support the publication of research results based on executable source code scripts (e.g., R or Python) and the underlying data.Hence, we did not consider workflow systems (e.g., Taverna (Wolstencroft et al. 2013)) or online repositories (e.g., Open Science Framework, https://osf.io/).Also, this paper does not discuss how to work reproducibly since this is covered already in literature (e.g., Rule et al. (2019), Sandve et al. (2013), Greenbaum et al. (2017), Markowetz (2015)).The review is a snapshot of the highly dynamic area of publishing infrastructures.Hence, some of the collected information might become outdated, e.g., an application might extend the set of functionalities or be discontinued.Still, reviewing the current state of the landscape to reflect on available options is helpful for publishers, editors, reviewers, authors, and librarians.All collected data is available in the supplements (see Data and Software Availability).The paper is structured as follows: First, we survey fundamental concepts and tools underlying the applications.We then introduce each application and the comparison criteria followed by the actual comparison.The paper concludes by a discussion about the observations we made, trends, and limitations.

Fundamental concepts and tools
Before reviewing applications for publishing reproducible research, we briefly survey fundamental concepts and tools that underpin the applications.This overview is needed to understand how the applications work and what the limitations are.

Packaging computational research reproducibly
The traditional research article alone is not sufficient to communicate a complex computational analysis (Donoho 2010).To address this issue, computational reproducibility concerns the publication of code and data underlying a research paper.This form of publishing research allows reviewers to verify the reported results and readers to reuse the materials (Barba 2018).To achieve that, all materials are needed, including not only the data and code but also the computational environment.A basic concept for such a collection is the research compendium, a "mechanism that combines text, data, and auxiliary software into a distributable and executable unit" (Gentleman and Lang 2007).The concept was extended by a description and snapshot of the software environment using containerization resulting in the executable research compendium (Nüst et al. 2017).Containerization and virtualization are mechanisms to capture the full software stack of a computational environment, including all software dependencies in a portable snapshot (Perkel 2019).In contrast to containerization, virtualization also includes the operating system kernel.Despite this difference, both approaches have proven to improve transparency and reproducibility (Boettiger 2015;Howe 2012).One containerization technology is Docker, which is based on so-called Dockerfiles, human and machine readable recipes to create the image of a virtual environment (Boettiger 2015).These recipes add an additional layer of documentation making Docker a popular tool in the area of computational reproducibility (Nüst and Hinz 2019).A research compendium should contain an entry point, i.e., a main file that needs to be executed to run the entire analysis.One option to realize these entry points is the concept of literate programming, an approach for interweaving source code and text in one notebook (Knuth 1984).Two popular realizations of such notebooks are Jupyter Notebooks (Kluyver et al. 2016) and R Markdown (Baumer et al. 2014).Combining source code and data in one document is advantageous over other approaches, such as having code scripts and the article separated, which might result in inconsistencies between the two.A further advantage is the possibility to execute the analysis with a single click, so called one-click-reproduce (Pebesma 2013).

Licensing and Citation
Appropriate licensing of research components is crucial yet complex, as copyright laws differ between component types, e.g., data, software, and text (Stodden 2009).This is particularly important when it comes to reusing research components, which is one of the main goals of research compendia.A further level of complexity emerges if research compendia include, for example, parts of the data and the code of several already published papers.A typical use case is reusing code of a specific version published in a repository, while the same code is developed and stored on a public repository (e.g., GitLab).Besides conscious handling of licenses and copyrights, building on top of the work of others requires adequate citations.This can be supported by connecting the research components with the help of metadata including permanent and global identifiers, e.g., DOIs (Stodden et al. 2016), which can be also used for data (Park and Wolfram 2019) and software (Fenner et al. 2016).

Ethical and technical issues
Frequently mentioned issues related to computational reproducibility concern sensitive data and large data files.To tackle the issue of sensitive data, a first step would be to anonymize the data.Another option is to involve a trustworthy authority which ensures that the results in the article can be achieved based on the used data (Pérignon et al. 2019).In this case, public access is not required.To ensure that these solutions are not exploited, authors should argue why hiding or providing synthetic data is required and reviewers can then decide whether the reasons are valid.A further solution is the concept of cloud-based data enclaves, which provide data access only to authorized persons (Foster 2017).Such approaches for access control could be connected with the applications discussed in this paper.
Large data files, e.g., global remote sensing datasets quickly reach several petabytes.However, a large number of papers are based on datasets that can be stored on public and free data repositories, such as Open Science Framework (file size limit only for individual files, https://help.osf.io/hc/en-us/articles/360019737894-FAQs#what-is-the-individual-file-size-limit) or Zenodo (max.50GB by default, extension possible, https://help.zenodo.org/whatsnew/).Further limiting factors are long computation times and the need for specialized hardware, such as high-performance computing clusters (Ahn et al. 2013).

Materials and Methods
To obtain an overview of what the applications supporting the publication of reproducible analyses provide as well as the trends and limitations, we compared them across a set of criteria.

Materials
To ensure that the stakeholders receive current recommendations, we considered an application as part of our analysis if (i) it was actively maintained at the time the data for this paper was collected (5th-13th Dec 2019), (ii) it supported publishing executable code and data which can be inspected and reused, and (iii) the application was explicitly connected to the publication process.Hence, we did not consider technologies that alone cannot support the publication process of code and data as further infrastructure is needed (e.g., Docker) or applications that only provide access to data or code (e.g., Zenodo).We found the applications during literature research and discussions at conferences or workshops.

Applications
Based on the sample criteria, ten applications were selected for the review.In the following, we briefly introduce them in alphabetical order.
Researchers having a repository (e.g., on GitHub/Lab, Zenodo) including, e.g., a Jupyter notebook can use Binder (https://mybinder.org/) to make it available in an executable environment (Jupyter et al. 2018).Readers can launch the analysis from a Binder-ready repository and inspect the workflow in a browser.Binder creates a containerized environment from a repository based on configuration files.In Code Ocean (Clyburne-Sherin, Fei, and Green 2019), authors can create so-called "capsules" which contain code, data, and the computational environment including the version of the operating system and dependencies.Readers can, while studying the article, execute and inspect the analysis in a separate window below the online version of the article or on Code Ocean's website.
The eLife Reproducible Document Stack (RDS, https://elifesciences.org/labs/b521cf4d/reproducible-document-stack-towards-a-scalable-solution-for-reproducible-articles) enables authors to publish executable documents based on Stencila (https://stenci.la/),an open-source editor for articles.The executable document, which contains the whole narrative and executable code snippets, is not only a supplement but the actual scientific article.Galaxy (Goecks et al. 2010) is a web-based application for developing computational analyses without programming expertise.Scientists can upload and analyze data by using Jupyter Notebooks (Grüning et al. 2017).Gigantum (https://gigantum.com/)builds on top of Git and packages code, data, the computational environment, and the work history into a Git repository.Gigantum is composed of a client application for creating as well as executing analyses locally, and a cloud-based infrastructure for sharing computations and collaborating with peers.Manuscripts (https:// www.manuscripts.io/about/) is an online tool for writing executable documents collaboratively based on the concept of literate programming, but featuring a "What you see is what you get" user interface.The runtime environment of the author is, however, not considered.o2r (Nüst et al. 2017) addresses publishers who want to extend their existing infrastructure by a reproducibility service during the process of paper submission (Nüst 2018).Authors can also create interactive figures, allowing reviewers and readers to check the robustness of the results, e.g., by changing model parameters using a slider (Konkol, Kray, and Suleiman 2019).REANA (Šimko et al. 2019;Chen et al. 2018) provides a formal specification to guide authors through the process of capturing input datasets, code, and the computational environment.Based on this structure and after creating some configuration files manually, REANA provides a set of command line interface (CLI) commands to run large analyses on a remote REANA cloud.ReproZip (Steeves, Rampin, and Chirigati 2017;Chirigati, Rampin, et al. 2016) provides a set of CLI commands for encapsulating data, code, and the computational environment automatically.Users can execute the resulting bundle on a server provided by ReproZip (Rampin et al. 2018) or locally on different computer systems.With Whole Tale (Brinckman et al. 2019), authors can create so called "Tales" that combine narrative, data, code, and the computational environment.Readers can inspect the materials and execute the analysis in the original environment.

Rationale for the comparison criteria
We identified the comparison criteria considering the needs of stakeholders of the scholarly publication process described by Nüst et al. (2017), i.e., those of publishers, editors, authors, reviewers, readers, and librarians.There is some overlap regarding stakeholder needs, for example, publishers as well as authors aim at attracting readers and providing a convenient reading experience for reviewers.
Publishers need to know whether they can integrate the application into their existing infrastructure.The applications can be either made available as open source tools for own hosting or as a service hosted by the provider.If the tool is available for free under an open license, publishers only have to consider costs for maintaining the infrastructure.Moreover, publishers gain full control and can customize the interface or processes according to their own specifications.In case of a paid service, publishers can take advantage of not being responsible for the maintenance.A further criterion relevant for publishers is the development stage of the application, i.e., if it was already used in published articles.
Editors of journals need to ensure that a service for publishing reproducible research is consistent with the tools the authors typically use and common practices in their scientific field.For example, journals regularly receiving submissions containing Jupyter Notebooks should not choose a service that supports only R Markdown.This aspect might also affect the author and reviewer guidelines, for which the editors are responsible.A further relevant aspect is the addressed research area.Some applications might address specific fields and thus provide features tailored to domain-specific requirements.
Authors need to submit research materials efficiently.Hence, we checked how authors can upload their files and which submission formats and programming languages are supported.We also considered which license submitted materials receive, since this is a frequently mentioned aspect of papers discussing reproducibility guidelines (Stodden et al. 2016).Although licensing is relevant for all stakeholders, authors are particularly responsible for taking care of it.We also checked whether the applications can deal with sensitive data.
For readers and reviewers, open reproducible research comes with several benefits, such as advanced search capabilities, re-running workflows, inspecting results in detail (i.e.looking at code or data files), modifying parameter settings, and reusing the data or the analysis for the own work (Konkol and Kray 2018).We thus checked whether the tools provide any specific support for such investigations of the research materials.
Librarians are tasked with preserving research materials.We checked how the materials are stored and shared, and if modifying or deleting them after publication is possible.
Based on these comparison criteria, we investigated the project websites, the actual applications, GitHub/Lab repositories, scientific articles (if available), and blog posts.Since most of the sources were not scientific articles, the supplements contain screenshots and URLs to show where we found the corresponding information.

Results
In the following, we compare the applications considering the needs of the stakeholders.Table 1 summarizes aspects relevant for publishers, i.e., if self-hosting is possible, which license is assigned to the application, whether it is already in use or in a beta stage, and the funding source.From the ten applications, eight allow self-hosting.Code Ocean and Gigantum provide the service themselves.eLife RDS, o2r, and REANA (in Table 1 marked by *) require own installations since no free online deployments exist.Three applications are released under the BSD-3-Clause License, three under MIT of which Gigantum assigned this license to the local tool and not to the cloud service, one under Apache 2.0, one under the CPAL 1.0, and one under Academic Free License 3.0.These licenses allow operators to host their own service as well as to modify the software according to their individual needs and styles.This means, however, they also have to maintain the infrastructure and provide the required technological resources as well as personnel.In contrast, Code Ocean's infrastructure and Gigantum's cloud service are provided in exchange for payment.From the reviewed applications, four are rather experimental and six are already in use as shown by the example papers with workflows based on the corresponding application.Seven applications receive funding from public or private science foundations.Code Ocean and Gigantum offer a commercial service.
Table 2 summarizes aspects relevant for editors and authors, i.e., the scientific domains, supported submission formats, upload mechanisms, and license terms.Although none of the investigated applications are strictly tied to a specific domain, we observed that some of them focus on particular areas.For example, Galaxy provides a rich set of features tailored to use cases in the life sciences.Other applications originate from a particular domain, e.g., eLife's RDS comes from the life sciences whereas REANA focuses on particle physics.From the ten applications, nine support literate programming approaches by default, e.g., Jupyter Notebook or R Markdown.Manuscripts supports Markdown, but also code execution via embedded Jupyter Notebooks.4 addresses libraries and other institutions with a mandate to preserve and provide access to research outputs.It includes information on how the research materials are stored and shared, and whether modifying or deleting content once published is possible.Five applications provide storage, though it remains unclear whether they run the servers by themselves or by third-party services, and what kind of backup and archiving is implemented.Seven applications give hosts the option to store research materials independently, e.g., on the publisher's infrastructure.The free available instance of Binder, MyBinder.org(https://mybinder.org/),stores Docker images temporarily but beyond that, no storage is provided.Whole Tale and o2r use existing long-term preservation services, e.g., Zenodo and DataOne.Regarding the possibility to modify or delete materials once published, we assigned "possible" if there is any way to do so.In Binder, REANA, and ReproZip, modifying/deleting content is possible if the research materials are stored on GitHub/Lab, but not when stored on Zenodo.The same is true for Galaxy, Gigantum, and Manuscripts, which allow users to edit/delete contents stored in the cloud.Code Ocean and Whole Tale assign DOIs to published contents making it impossible to edit these after publication.The same applies to o2r but only if the materials are archived.Finally, in eLife's RDS, the article is composed of text and code.Deleting it is thus equivalent to withdrawing a paper.All in all, seven applications allow modifying published materials.However, this issue is mitigated when researchers "go the extra mile" and also publish their materials in long-term repositories, such as Zenodo.One exception is Code Ocean which allows modifications (no deletions) but assigns a new DOI to the modified content.Finally, authors need a way to share reproducible results in their paper.This is possible via a URL/DOI to the application (eight applications provide this possibility) or a URL to an online repository (2).

Discussion
Several projects develop applications for publishing computational research.One might think the applications, since they all strive for the same overall goal, resemble each other.However, the overview in this paper (see Tables 1-4 ) shows that the applications address different issues and needs.This increases the chances for stakeholders to find a suitable application for their individual requirements.

Needs of stakeholders
Publishers: A critical decision is whether publishers want to host an infrastructure by themselves or engage a provider.Applications exist for both approaches though the majority of them allow self-hosting.Accordingly, all self-hosting solutions have an open license enabling operators to create customized versions of the platforms and the peer review process.A further advantage is the mitigation of risks regarding vendor lock-in or grant-based projects which expire at some point.
Nevertheless, it remains unclear which costs publishers have to expect when hosting an infrastructure.
The final costs strongly depend on the number of views and execution attempts, workflow sizes, and manipulation options.These parameters differ between use cases and could be the basis for future research, e.g., on stress tests.Therefore, the metrics of existing publications might provide first insights to calculate the required resources.The Binder instance MyBinder.orgprovides an initial estimate regarding costs (https://mybinder.org/v2/gh/jupyterhub/binder-billing/master?urlpath=lab/tree/analyze_data.ipynb).However, further data on infrastructure costs from the other services would help to calculate costs in a more accurate way, albeit this transparency is only realistic for non-profit projects.
Working with applications that are already in use and those in a beta stage can both have advantages and disadvantages.While applications already in use provide initial evidence that they work, it might take more effort to make adjustments in order to fit the publisher's infrastructure.In contrast, applications in a beta stage can adjust their feature set and consider new contributions without worrying about running instances and backwards compatibility, but the deployment of the applications might reveal new issues.

Editors and authors:
The research area does not narrow down the number of options.Although some applications come from specific domains, e.g., the life sciences, none of them is restricted to a specific field.Regarding submission formats, there is a trend towards literate programming approaches.Most applications either support Jupyter Notebook or R Markdown which both have proven to support reproducibility (Grüning et al. 2018).However, some journals and publishers have particular requirements, e.g., they rely on LaTeX.Since transformations to other document types are often cumbersome and shifting author requirements can be a lengthy process, it might be easier to have reproducible documents as a supplement, potentially for a transition period until the executable documents are widely accepted.Nevertheless, eLife's RDS shows that a scientific article combining executable code with narrative is possible today and comes with advantages with respect to communicating scientific results, e.g., studying text and analysis in parallel while also being able to manipulate the analysis.
A critical issue is that not all applications explicitly handle the copyright of the shared materials.Those who do, fortunately, either require or encourage open licenses.Licensing is important to enable reusability and thus a recommendation mentioned frequently in papers discussing reproducibility guidelines (Nosek et al. 2015;Stodden 2009).Therefore, the platforms should inform users about licenses, e.g., by referring to existing advising resources (e.g., https://choosealicense.com/) and ideally require open licenses.
A further limitation of the applications is that the anonymity of the authors is not guaranteed during the review process.All applications require an account for creating reproducible results and the name of the creator is usually visible making double-blind review impossible.However, access to code and data is particularly important for reviewers, since they decide on accepting or rejecting a submission.One solution might be to create an anonymous version of the materials, as it is possible with Open Science Framework (https://help.osf.io/hc/en-us/articles/360019930333-Create-a-View-only-Link-for-a-Project) or to adopt an open peer-review process.

Reviewers and readers:
Being able to reproduce computational results in a paper is a clear benefit.However, open reproducible research comes with a number of further incentives with respect to finding and inspecting papers (Munafò et al. 2017).Most search tools provided by the applications do not take full advantage of the information contained in code and data files, e.g., spatiotemporal properties.Instead, they either provide a keyword-based search or no search at all.For inspecting materials, most solutions either provide their own UI or integrate a development environment, e.g., JupyterLab, RStudio.In both cases, users can directly access, manipulate, and reuse the code.However, readers might still need to understand complex code scripts.Moreover, identifying specific parameters buried in the code and finding out how to change these can be a daunting task.The concept of nano-publications (Kuhn et al. 2016) or bindings (Konkol, Kray, and Suleiman 2019) might help to solve these issues.A further need in this context is a UI for comparing original and manipulated figures since differences in the figure after changing parameters might be difficult to spot.Most applications do not provide any support for substituting research components, e.g., the input datasets.This might be due to the plethora of complex interoperability issues with respect to data formats or column names in tabular data.Only ReproZip (Chirigati, Rampin, et al. 2016) and o2r (Konkol and Kray 2018) provide basic means to substitute input datasets, yet they require users to ensure compatibility.

Librarians:
The state of the research materials is an essential issue when it comes to publication.While some applications fix the state of the research materials by assigning a DOI and archiving a snapshot, others allow changing and deleting them.This is a disadvantage with respect to reproducibility since verifiability and accessibility are not given anymore.In addition, if self-hosting is not possible, the computational analysis of an article will be executable only as long as the project and its infrastructure exist.This dependence is a crucial aspect with respect to archiving.A further dependence is the fundamental technology.Without Docker, a Dockerfile cannot be run anymore but it is still readable and provides important machine-and human-readable information on how to run the analysis.This is also true for source code scripts which are plain text files and thus can be opened using any editor.These examples demonstrate the importance of using open and text-based instead of proprietary and binary file formats in science.Due to these issues, researchers should consider archiving the research materials on platforms, such as Zenodo in addition to an executable version (which should be the same version) using one of the applications.

Limitations
This work is subject to a number of limitations.The scope of the paper is narrow and does not cover all applications that support the publication of computational research (e.g., workflow system, such as Taverna).
In addition, the information in this review paper might become outdated quickly but having a structured overview can still be helpful for the involved stakeholders to decide on a technology.Finally, the properties we investigated in this survey are certainly not complete.Still, stakeholders requiring more information can use the overview as a starting point for further research.

Conclusions
Open science is on the rise and obtains more attention and expressions of support from all stakeholders including publishers, editors, reviewers, authors, readers, and institutions responsible for archiving, e.g., libraries.Despite these developments, publishing computational results reproducibly is still a challenge for all parties involved in scholarly communication.Fortunately, several projects aim at tackling these issues by designing applications to support the publication of executable research results.The key contribution of this paper is an overview of these applications, their commonalities, and differences.This overview can be used as a decision support for publishers facing the question of whether to host an application by themselves, for editors who want to ensure that the application is conform with the author and reviewer guidelines, for authors who want to create reproducible analyses efficiently, and for reviewers who want to verify the results and to suggest potential solutions during the review process.Moreover, the overview considers the needs of readers who want to better understand and reuse research materials, such as code scripts and data.Finally, the survey contains aspects relevant for archiving.The applications all provide a rich set of functionalities and address many reproducibility issues.However, issues related to ethics, privacy, big data, and long computation times are not fully solved, yet.Beyond these challenges, considering executable submissions already during the review process comes with a number of novel research questions: How many reviewers try to reproduce submissions using one of the applications?How does a reproducibility attempt (whether it failed or succeeded) affect a reviewer's decision?Is the effort required for reviewing reproducible papers the same as for a traditional article (i.e., only reading the text), or will reviewers refuse reviews since they fear additional work?And finally, how much time does it take to review a paper supplemented by reproducible documents and how much additional understanding do reviewers obtain?These quantitative (time, user interactions) and qualitative (interviews, questionnaires) measures can help to improve the applications and eventually foster the success of open reproducible research.

Table 1 :
Overview of properties relevant for publishers, i.e., if self-hosting is possible (* denotes only self-hosting is possible), which license the applications have, the stage of the project (in use or beta), and the funding source.Seven applications are extensible and provide the possibility to configure the application to support further submission formats or programming languages.Except for Code Ocean which also supports MATLAB and Stata, all applications only support non-proprietary programming languages.For making code and data available on the platform, five applications provide file upload.Five applications provide the possibility to upload materials via an external cloud or repository, e.g., Zenodo.However, uploading materials might be disadvantageous for papers based on large data files.For these cases, eLife's RDS (based on Stencila), REANA, and ReproZip allow local usage.Researchers can also work locally with Gigantum, but then need to synchronize with the online service to access all features.Despite the importance of licensing, we could not find information on copyright for research materials in four applications.Whole Tale and Gigantum only allow open licenses whereas Code Ocean, Galaxy, and o2r encourage it.eLife assigns an open license to the article text only.

Table 2 :
Overview of aspects relevant for editors and authors, i.e., the addressed research area, which submission formats are supported, how authors can upload materials, and copyright.Table3summarizes aspects relevant for reviewers and readers.From the ten applications, five provide a keyword-based search for papers whereas five do not provide any search feature.o2r provides a spatiotemporal search combined with thematic properties, such as libraries used in the code.Nine applications provide tools for inspecting code and data, six of them by providing an own user interface (UI) and three by embedding a programming environment (e.g., JupyterLab, RStudio).Though REANA does not provide supportive tools for inspection, the materials can be viewed when stored on public repositories, e.g., GitLab.Nine applications provide tools for downloading materials.Projects created with REANA can be downloaded if stored on public repositories which already provide a download functionality.Eight applications allow readers to execute the analysis in the browser on a remote server.Gigantum provides a UI running locally, REANA projects are executed via the CLI in a remote REANA cloud.Each application allows manipulating the code and rerunning it based on a new parameter.Most commonly, users can directly manipulate the code in the browser (6 applications provide this option) or locally (Gigantum).In REANA, users can pass new parameter values via the CLI, in ReproZip via the CLI or input fields using ReproServer.The o2r platform allows authors to configure UI widgets giving reviewers/readers the chance to interactively manipulate parameter values, e.g., by using a slider to change a model parameter within a certain range.

Table 3 :
Overview of features relevant for reviewers and readers, i.e., searching for papers and materials, inspecting code and data, downloading materials, executing the analysis, and manipulating the code.

Table 4 :
Overview of properties relevant for long-term preservation, i.e., how the research materials are stored, if they can be modified/deleted after publication, and how they can be shared in articles.