Choosing the Right Storage
The best storage option depends mainly on how your workflow accesses the data.
Before choosing a location, consider whether your data needs fast filesystem access, object-based access, or only temporary storage during a compute job.
Warning
Perform regular backups of your data to a safe location. The ML Cloud does not perform data backups.
Start Here
Choose storage based on how your workflow needs to access the data.
flowchart TD
A([How will your workflow access the data?])
A --> B{Directly through files<br/>and directories?}
B -->|Yes| C{Which cluster<br/>are you using?}
C -->|Galvani| D[Lustre]
C -->|Ferranti| E[Weka]
B -->|No| F{Through an<br/>S3-compatible application?}
F -->|Yes| G[Ceph S3]
F -->|No| H{Needed only while<br/>a compute job is running?}
H -->|Yes| I[Local node storage]
H -->|No| J[Contact the ML Cloud team]
D --> K[Active data used by<br/>jobs on Galvani]
E --> L[Active data used by<br/>jobs on Ferranti]
G --> M[Object-based workflows<br/>and bucket access]
I --> N[Temporary files, caches,<br/>and intermediate data]Before Creating Another Copy
Check whether the data is already available as a shared dataset.
Quick Comparison
| Storage option | Access method | Suitable for |
|---|---|---|
| Lustre on Galvani | Mounted filesystem | Active data used directly by jobs on Galvani |
| Weka on Ferranti | Mounted filesystem | Active data used directly by jobs on Ferranti |
| Ceph S3 | S3-compatible tools and applications | Object-based workflows and data that does not require mounted filesystem access |
| Local node storage | Filesystem available during a job | Temporary files, caches, and intermediate processing data |
Lustre and Weka
Lustre and Weka are the high-performance filesystems used on Galvani and Ferranti.
Use them when your workflow:
- requires normal file and directory access
- reads or writes data repeatedly during compute jobs
- processes many files
- uses tools that expect mounted filesystem paths
- needs to share active project data within a group
Use Lustre when working on Galvani and Weka when working on Ferranti.
High-performance storage should primarily contain data that is actively being used. Temporary files, unnecessary copies, and outputs that can easily be reproduced should be removed when they are no longer needed.
For cluster-specific paths and instructions, see:
Ceph S3
Ceph S3 is object storage. Data is stored in buckets and accessed through S3-compatible tools or applications rather than through a normal mounted directory.
Consider Ceph S3 when:
- your application supports S3 access
- your workflow is designed around objects or buckets
- the data does not need to be accessed repeatedly through normal filesystem operations
- data needs to be transferred or shared through bucket permissions
- you are storing large collections that do not require direct high-performance filesystem access
Ceph S3 may be less suitable for workflows that repeatedly open many small files using standard filesystem operations.
To set up and use Ceph S3, refer to the documentation for Galvani and Ferranti
Local Node Storage
Local node storage is temporary storage available on an individual compute node during a job.
It can be useful for:
- temporary files
- caches
- unpacked archives
- intermediate processing data
- reducing repeated reads from shared storage during a job
Files on local node storage are not intended for long-term retention and may disappear when the job ends or the node is cleaned.
Copy all important outputs back to Lustre, Weka, or another persistent storage location before the job finishes.
Storage changes throughout projects
The most appropriate storage location may change during a research project.
For example, a dataset may be stored on Lustre or Weka while it is actively processed. Temporary files may be written to local node storage during jobs. Once processing is complete, unnecessary intermediate files and duplicate copies can be removed. Relevant data and outputs may be moved to Ceph S3 for long-term storage at the end of a project.
Review your stored data regularly and ask:
- Is this data still actively used?
- Does it need high-performance access?
- Is another complete copy necessary?
- Can temporary or reproducible files be deleted?
- Is the dataset already available centrally?
Best practices
- Choose storage based on the way the data will be accessed
- Avoid unnecessary copies of large datasets
- Keep source data, derived data, and results separate
- Document where important project data is stored
- Review storage needs throughout the project
Not Sure Which Storage to Use?
Contact us through the ticketing system. Include the cluster, approximate data size, file structure, expected access pattern, number of users, and how long the data needs to be retained.
Created: July 27, 2026