Skip to content

Reproducibility

Reproducible research allows you and others to understand how results were produced and, where possible, repeat the same analysis.

A reproducible workflow requires more than code. The data version, software environment, parameters, and processing steps may all affect the results.

Record the Research Workflow

Document the main steps between the original data and the final result.

Record information such as:

  • the data source and version
  • preprocessing and quality-control steps
  • model or analysis settings
  • software and package versions
  • relevant random seeds
  • commands used to run the workflow
  • the location of important outputs

Document Decisions as You Work

Recording versions, parameters, and processing decisions during the project is much easier than reconstructing them later.

Keep Code Under Version Control

Use a version-control system such as GitLab to track changes to code, configuration files, environment specifications, and documentation.

Avoid relying on filenames such as:

analysis_final.py
analysis_final_new.py
analysis_final_really_final.py

Version control provides a clearer record of what changed, when it changed, and why. Use meaningful commit messages and identify important project versions with tags or releases where appropriate.

Do Not Store Secrets in GitLab

Do not commit passwords, access tokens, SSH keys, S3 credentials, or other sensitive information.

Record the Software Environment

A workflow may behave differently when software or dependency versions change.

On the ML Cloud, Singularity containers can be used to package the software and dependencies required for a project. This makes it easier to reuse the same environment across jobs, share it with collaborators, and return to an analysis later.

For reproducibility, document:

  • which container image was used
  • the version or date of the image
  • where the image or definition file is stored
  • whether GPU support is required
  • any important additional dependencies

For technical instructions, see the Singularity Containers tutorial.

Keep the Environment with the Project

Keep the container definition file, or a clear reference to the image, together with the project documentation.

Track Data Versions and Provenance

Code can only reproduce a result when the correct input data is available.

Record:

  • the dataset name and original source
  • the version or release date
  • the storage location
  • any transformations applied
  • the relationship between original and derived data
  • relevant licence or access restrictions

For shared datasets, reference the centrally available dataset rather than creating an undocumented copy.

Do not overwrite original data during preprocessing. Store derived data separately and document how it was created.

Configuration files and experiment settings

Keep important parameters separate from the main analysis code where possible.

Configuration files can record:

  • dataset paths
  • model settings
  • learning rates and batch sizes
  • preprocessing options
  • random seeds
  • output locations
  • resource requirements

This makes individual runs easier to compare and reduces the need to modify code for each experiment.

A project could, for example, contain:

project/
├── README.md
├── code/
├── config/
├── data/
├── logs/
└── results/
Recording Slurm jobs and experiments

Keep the Slurm job script together with the code and configuration used for an experiment.

Where relevant, record:

  • the job ID and cluster
  • requested CPUs, memory, and GPUs
  • the container or software environment
  • the input data version
  • the configuration file
  • log files and output location

Use clear run names or separate directories and avoid overwriting previous results.

Randomness and repeatability

Machine-learning and statistical workflows often include random processes.

Where possible:

  • set and record random seeds
  • record data splits
  • note whether deterministic execution was enabled
  • repeat experiments when results may vary
  • report variability rather than only one run

A fixed seed improves repeatability but may not guarantee identical results across different hardware or software versions.

Checksums and data integrity

Checksums can help confirm that files have not changed during transfer or preservation.

For example:

sha256sum <filename>

Checksums verify file integrity, but they do not replace documentation, metadata, or version information.

Automating repeated steps

Manual steps are easy to forget and difficult for others to repeat.

Where practical, use scripts or workflow tools for:

  • preprocessing
  • training
  • evaluation
  • creating tables and figures
  • exporting final results

The workflow should make the order of operations clear and minimise undocumented manual changes.

Preparing a project for reuse

Before sharing, publishing, or handing over a project, check that it includes:

  • an up-to-date README
  • the relevant code version
  • environment or container information
  • configuration files
  • data source and version information
  • instructions for running the workflow
  • expected inputs and outputs
  • licence and citation information
  • known limitations

Where possible, test the instructions in a clean environment to identify undocumented dependencies.

Best Practices

To improve reproducibility:

  • keep code under version control
  • record dataset versions and provenance
  • keep original and derived data separate
  • document the software environment
  • store parameters in configuration files
  • retain important job scripts and logs
  • set and record random seeds
  • avoid overwriting previous experiments
  • document manual decisions and exceptions
  • test whether the workflow can be run again

Need Help Making a Workflow Reproducible?

Contact us through the ticketing system. Include the cluster, project workflow, software environment, and the parts of the process you would like to reproduce or automate.

Last update: July 27, 2026
Created: July 27, 2026