Author Instructions For OJSP Reproducibility Review

Code Ocean is a cloud-based platform that facilitates reproducibility of computational research and supports reproducibility reviews. Authors can set up a private “compute capsule” on Code Ocean to which they can upload code and data and configure the computational environment required to run their code. Research reproducibility reviewers can access and run the compute capsule on the Code Ocean website to assess the validity of computational results presented in a manuscript, and share their assessment with editorial board members. Upon publication of the manuscript, the Code Ocean capsule is also published, assigned a DOI (digital object identifier) and made available to readers via a link/citation from the published article. OJSP is partnering with Code Ocean for this pilot because of the convenience and flexibility it offers to readers and reviewers, allowing them to readily reproduce computational work without having to deal with the overhead of setting up and configuring the computational environment. Code Ocean supports all open source programming languages, as well as Stata and MATLAB, and code and data can be imported in a number of different ways, including from a GitHub repository.

• Please read and follow the instructions on the Code Ocean website to set up a capsule for the reproducibility review of computational results presented in your OJSP manuscript. If necessary, Code Ocean staff are available to assist in setting up a compute capsule.
• The module should include the learned model that was used to obtain the computational results in your paper, and code that makes use of the model to generate results in the same manner in which you generated results for your OJSP manuscript. If input data is required to generate these results, this data should also be provided (either the entire dataset or an illustrative sample).
• For review purposes, create a private link to your capsule and provide that link to the OJSP administrator (Rebecca Wollman, r.wollman@ieee.org) once your article has received an AQR decision indicating that it is accepted for publication subject to passing a reproducibility review. Please set up the capsule directly after submission of your manuscript submission to avoid delaying the review and publication process.You will be required to provide the private link for reproducibility review within 48 hours of receipt of an AQR decision notification.
• You are strongly encouraged to read the FAQ below, which are intended to answer common questions regarding the reproducibility review.

Author FAQ for OJSP Reproducibility Reviews

1. To what degree is the Code Ocean capsule provided for my manuscript expected to reproduce the results in my paper?

Please see the instructions for reproducibility reviewers provided at “Reviewer Instructions for OJSP Reproducibility Reviews”, on this page, which set the expectations for the level of reproducibility required from the Code Ocean code. Among other things, situations that require special consideration are outlined in those instructions . As we note there: “The goal of the reproducibility review is to perform a good faith validation of the results. It is not intended to be an adversarial attempt to find unusual conditions under which the code fails.”

2. What material is provided to reproducibility reviewers, and is confidentiality maintained for the reproducibility review?

In addition to access to the Code Ocean capsule, the manuscript and the AQR notification will be provided to the reproducibility reviewers. The reproducibility review and, specifically, the use of the Code Ocean platform for peer review of code associated with this manuscript, is subject to the same confidentiality agreements as the rest of the review process.

3. What happens to the Code Ocean module once my OJSP article is published?

Upon acceptance and publication of your OJSP article, the Code Ocean module you have submitted for reproducibility review is also made publicly available and assigned a DOI (digital object identifier). The availability of code associated with your article on Code Ocean will be advertised on the IEEE Xplore page for your OJSP article, with a badge indicating that reviewed code is available, and via a link to the published Code Ocean capsule.

4. Will the reproducibility review require revisions to my OJSP manuscript?

Normally, we do not expect the reproducibility review process to result in any revisions to the manuscript. Minor corrections/clarifications that the reproducibility reviewers suggest for your manuscript may be incorporated in the final submission materials you provide for your manuscript, as per the guidance of the Associate Editor/Research Reproducibility Editor.

5. Why is OJSP using Code Ocean and not popular code sharing platforms such as GitHub?

While source code published on platforms such as GitHub is helpful for researchers, a substantial amount of time and energy is usually required for a researcher to set up such source code to run in one’s own computational environment. Code Ocean uses industry-standard “Docker Containers” for their compute capsules, which, in addition to providing the code, also encapsulates the computational environment, allowing for it to be replicated exactly. The Docker containers can also be exported from Code Ocean and replicated externally, outside of Code Ocean. Importantly, the use of Code Ocean substantially simplifies and streamlines the work for reproducibility reviewers, allowing them to do all their tasks on the Code Ocean platform itself. This is critical for maintaining timeliness of the review process.

6. What license does Code Ocean require?

   Code Ocean has a flexible license policy allowing you to choose between several common license options for your compute capsule, or to upload your own license file, should you so choose.

7. I have questions about the reproducibility review that are not addressed in the information above, who should I contact?

If you have questions regarding the OJSP research reproducibility review process, please contact either the OJSP EiC (Brendt Wohlberg, ojsp-eic@ieee.org) or the OJSP Research Reproducibility Editor (Gaurav Sharma, ojsp-rre@ieee.org). Note that for specific detailed questions regarding the Code Ocean platform, we recommend that you contact Code Ocean support for a quicker response, and for purely status queries, please contact the OJSP administrator, (Rebecca Wollman, r.wollman@ieee.org).