Skip to content

Research Data Management on the ML Cloud

Research data management helps you keep data organised, understandable, secure, and reusable throughout your work on the ML Cloud.

Research data may include downloaded or collected datasets, simulation outputs, processed data, model checkpoints, configuration files, logs, and analysis results. Managing these files carefully makes collaboration easier, reduces unnecessary duplication, and helps others understand how results were produced.

This page provides practical guidance for working with research data on the ML Cloud.

The Reasearch Data Lifecycle

Research Data Lifecycle

The stages seen above often overlap and may be revisted throughout a project.

Before you start

Data management decisions should ideally be made before data is uploaded or generated.

Consider the following questions:

  • What data will the project use or produce?
  • How much storage will be required?
  • Does the data already exist on the ML Cloud or within your group?
  • Is the data public, internal, confidential, personal, or otherwise restricted?
  • Who is responsible for the data?
  • Who needs read or write access?
  • Which outputs must be preserved?
  • Which files can be reproduced and safely removed later?

For important or shared datasets, identify a responsible person or group. This responsibility may include managing access, documentation, maintenance, and eventual handover.

Before storing personal, confidential, licensed, or otherwise restricted data on the ML Cloud, confirm that the planned storage, processing, and sharing arrangements meet the relevant legal, ethical, contractual, and institutional requirements.

Tip

Use the DFG checklist for planning and describing the handling of research data as a guideline when planning your project. It can help you consider data volume, documentation, storage, access, responsibilities, and long-term reuse.

Bring data onto the ML Cloud and store it

Record the source, date, and version of data that you download, receive, or generate. Where applicable, also record its licence and access conditions.

The ML Cloud provides different storage systems for different purposes. The best location depends on whether the data is actively used, temporarily required for a computation, shared with collaborators, or retained for future use.

In general:

Storage type Typical use
Home storage Small personal files, configuration files, and scripts
Work or project storage Active research data, environments, models, and results
Scratch or node-local storage Temporary files used during computation
Ceph S3 object storage Data that does not need to remain directly on an active cluster filesystem
GitLab Code, scripts, configuration files, documentation, and small text-based files
Nextcloud Documents and files used for convenient access or collaboration

The available storage systems and their exact behaviour differ between Galvani and Ferranti. Check the relevant cluster documentation before choosing a location.

Keep data copies under control

For important datasets:

  • keep one clearly identified authoritative copy
  • separate original, processed, and temporary data
  • document how derived or staged copies were created
  • avoid repeated downloads and undocumented duplicates
  • remove temporary copies when they are no longer needed
  • check that important transfers completed successfully

Temporary storage must never contain the only copy of important data.

Process, organise, and document data

Use a clear project structure so that source data, processed data, temporary files, and results can be distinguished easily.

For example:

project-name/
├── README.md
├── data/
│   ├── raw/
│   ├── processed/
│   └── temporary/
├── code/
├── configs/
├── models/
├── results/
└── logs/

This is only an example. Choose a structure that fits your project and document what each directory contains.

Recommended practices
  • Keep original data separate from processed data
  • Avoid modifying or overwriting original files
  • Store temporary data in clearly identified locations
  • Use descriptive file and directory names
  • Avoid unclear names such as final, final2, or new_final
  • Distinguish successful, incomplete, and failed outputs
  • Remove obsolete duplicates
Describe important datasets

Metadata is information that describes data, such as its title, creator, version, format, origin, and licence. Even simple metadata makes data easier to find and understand.

Have you recorded the following?

  • the dataset name and purpose
  • its source or creator
  • the date it was obtained or generated
  • the version used
  • the file formats and directory structure
  • the licence or conditions of use
  • relevant preprocessing or filtering
  • known limitations
  • the responsible person or group
  • related code, documentation, or publications

A Git repository is useful for code, documentation, and configuration files, but it does not replace proper management of the research data itself.

Keep data, code, and processing connected

For important analyses, simulations, or training runs, record:

  • the input dataset and version
  • preprocessing and analysis scripts
  • configuration files and parameters
  • software and library versions
  • environments or containers
  • Slurm job scripts and job IDs
  • relevant logs and error messages
  • random seeds, where relevant
  • output locations and result versions

Provenance is the record of where data came from and what happened to it.

Whenever possible, use scripts for important processing steps rather than relying only on commands entered interactively. Keep enough information to understand how an important result was produced and, where necessary, reproduce the process.

Share and collaborate responsibly

Before sharing data, check:

  • whether you are allowed to share it
  • whether its licence permits redistribution
  • whether it contains confidential, personal, or restricted information
  • who needs access
  • whether they require read-only or write access
  • whether the documentation is sufficient for others to understand the data
  • who will remain responsible for the shared copy

Use the least amount of access necessary. Avoid making directories broadly writable when collaborators only need to read the files.

Different sharing methods serve different purposes. Filesystem permissions, Access Control Lists, Nextcloud shares, Ceph S3 access, temporary links, and public repositories are not interchangeable.

Never share passwords, SSH private keys, or personal access credentials.

Protect important data

ML Cloud storage should not automatically be treated as a backup.

Decide which data is:

  • irreplaceable
  • expensive or time-consuming to reproduce
  • required to verify results
  • needed by collaborators
  • subject to a retention requirement
  • temporary and safe to remove

Important data should have a separate backup in an appropriate location. Where necessary, preserve the documentation, code, configurations, and environment information needed to understand it.

Protect data integrity by checking important transfers, avoiding accidental overwriting, and documenting incomplete or failed processing.

Complete your project and prepare data for reuse

Research data management continues when a project ends or when a researcher leaves a group.

Before completing or handing over a project:

  • identify the authoritative or final dataset versions
  • remove unnecessary temporary files and obsolete duplicates
  • document important data and results
  • preserve relevant code, configurations, environments, and logs
  • confirm who will remain responsible for the data
  • transfer ownership and access where necessary
  • confirm that collaborators can still access required files
  • decide which data should be retained, archived, published, or deleted
  • maintain any required backup outside the active cluster filesystem

Not all generated data needs to be retained indefinitely. Temporary outputs that can be recreated may be removed, while irreplaceable source data, important results, and data supporting publications may need to be preserved.

Review the ML Cloud offboarding guidance before an account or group membership ends.

FAIR in practice

The practices described above support the FAIR principles:

  • Findable: Data has a clear name, location, description, and version.
  • Accessible: The conditions and method for accessing the data are documented.
  • Interoperable: Data uses suitable formats, structures, and descriptions.
  • Reusable: Data includes documentation, provenance, licences, and known limitations.

FAIR does not mean that all data must be publicly available. Restricted data can still follow FAIR principles when its existence, documentation, access conditions, and responsibilities are clear.

See the related ML Cloud documentation for technical instructions on:

For any help with storage selection, data movement, permissions, shared datasets, documentation, provenance, backup planning, or project handover submit a ticket.

Last update: July 21, 2026
Created: July 16, 2026