Available Storage in Galvani

This page describes how Galvani deals with data storage. Right now the storage solution for Galvani and Galvani is the same.

On Storage Availability and Backups

Please note that storage space and services on Galvani is provided as is. We cannot guarantee high availability and back-up of any of our storage space, which refers to all storage on the ML Cloud.

Storage Quotas

Storage is a shared and limited resource and in a number of places we need to enforce quota to avoid that some script accidentally fills up the disk and the system becomes unusable for everybody.

Storage quota is specified in:

• Space limit: affects the aggregated size of all your files or files of a group. When this limit is reached you or the group cannot store more data (new data or increasing file sizes) on the system.

Quota applies to specific folders

Often it is intended that storage quota applies to a specific folder on the file system. For example, the so-called HOME quota shall apply to your home folder $HOME.

Quota applies to snapshots

It is important to remember that the file system snapshots that are created impact your storage quota. This means that if you truncate and then delete a file, the reduce in quota will not be visible immediately, but it will have a delayed effect of about 14 days.

QB Storage

The main storage backend for your data in Galvani is a Quobyte storage cluster. The only things that are not stored on the backend are the root disks of VMs and bare-metal machines as wells as ephemeral disks of VMs.

The following table summarizes the different storage options:

Table 2.1 ML Cloud File Systems For Galvani

QB Quotas and Freeing Space

The shared network volumes which provide $HOME, $WORK and /mnt/qb/datasets have user or group quotas in Galvani. You can query them with qinfo quota if your current working directory is part of those volumes.

There are two ways to delete/remove files and free quota:

• rm file: although this command will romove the file, the quota size decrease will be reflected with a delay of app. 2 weeks if files were present at the latest QB metadata snapshot.
• To free quota instantaneously use:

: > file
rm file

for a single file. For deleting a directory recursively use:

find DIRECTORY -type f ! -size 0c | parallel -X --progress truncate -s0
rm -rf DIRECTORY

• using mv file /ouside/of/volume/ might not free quota instantaneously. Instead use:

cp file /ouside/of/volume/
: > file
rm file