Singularity Containers
To ensure your results are reproducible and verifiable, use containers to package your research projects. Containers include an entire operating system, GPU libraries, etc., so this page discusses what a container is and how to work with our Singularity container system.
Docker images are not secure because they provide a means to gain root access to the system they are running on. This is not a problem because you can use Singularity which is an alternative to Docker that is both secure and designed for high-performance computing. Singularity is compatible with all Docker images and it can be used with GPUs and MPI applications, as well as Infiniband networks.
Singularity is already installed globally on all our systems, and should be immediately available on your command line. You can check the singularity version:
singularity --version
GPU support for containers
Commands that run, or otherwise execute (singularity) containers (shell, exec) can take an --nv option, which will setup the container’s environment to use an NVIDIA GPU and the basic CUDA libraries to run a CUDA enabled application. The --nv flag will:
- Ensure that the
/dev/nvidiaXdevice entries are available inside the container, so that the GPU cards in the host are accessible. - Locate and bind the basic CUDA libraries from the host into the container, so that they are available to the container, and match the kernel GPU driver on the host.
- Set the
LD_LIBRARY_PATHinside the container so that the bound-in version of the CUDA libraries are used by applications run inside the container.
Converting Docker Image to Singularity
If you want to convert a docker image called ubuntu with the tag latest to a singularity file called IMAGE_NAME.sif, just run this command:
singularity build IMAGE_NAME.sif docker://ubuntu:latest
You can execute this command on the login nodes, as well as on any of the compute nodes of the clusters. Also, you can find more information here.
Use docker2singularity
If you have access to a computer with docker installed (and running) you can also use the singularity2docker tool for converting your images. To convert the ubuntu image to singularity (as above) use:
docker run -v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp/test:/output \
--privileged -t --rm \
quay.io/singularity/docker2singularity \
ubuntu:latest
*.simg file you can then transfer to a ML-Cloud server. By itself this is more complicated then the first method, but it can become necessary if you want to make adjustements to a container that are difficult to reproduce in a singularity recipe.
You can find more about singularity2docker usage on github.
Build Singularity on ML Cloud Example
You can build a customer image by using one of the published images as base image.
First, Set the cache and tmp directories for singularity. This is required to build the container, as otherwise storage can be insufficient and result in a build failure.
export SINGULARITY_CACHEDIR=#SCRATCH/group-name/your-username
export SINGULARITY_TMPDIR=#SCRATCH/group-name/your-username
Mounting behavior
Singularity by default mounts these 5 folders from the host filesystem into the container filesystem:
/home/$USER
/tmp
/dev
/sys
/proc
so if your script or data is in one of these folders then you don't need manual mounting and can simply specify the path to the files to singularity commands.
Creating a definition file
To build an image , you need to create a file called definition file. This is a file with some instructions telling Singularity which image should be used as base image and which commands you would like to run. Then execute:
singularity build --fakeroot your-custom-image.sif def-file-example.de
where def-file-example.def if the definition file. To build an image without root privileges, use --fakeroot.
See Example Definition Files
Cuda with Conda
BootStrap: docker
From: nvcr.io/nvidia/cuda:12.5.0-devel-ubuntu22.04
%post
# Downloads the latest package lists.
apt-get update -y
# Install Python3 requirements
# --> python3-tk is required by matplotlib.
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
python3 \
python3-tk \
python3-pip \
python3-distutils \
python3-setuptools \
build-essential \
wget \
bzip2 \
ca-certificates \
libglib2.0-0 \
libxext6 \
libsm6 \
libxrender1 \
git
# Reduce the size of the image by deleting the package lists we downloaded,
# which are useless now.
apt-get -y clean
rm -rf /var/lib/apt/lists/*
#Installing Anaconda 3
wget -c https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
/bin/bash Anaconda3-2020.02-Linux-x86_64.sh -bfp /usr/local
#Conda configuration of channels from .condarc file
conda config --file /.condarc --add channels defaults
conda config --file /.condarc --add channels conda-forge
conda update conda
#List installed environments
conda list
Pytorch with conda
BootStrap: docker
From: nvcr.io/nvidia/pytorch:24.05-py3
%post
# Downloads the latest package lists.
apt-get update -y
# Install Python3 requirements
# --> python3-tk is required by matplotlib.
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
python3 \
python3-tk \
python3-pip \
python3-distutils \
python3-setuptools \
build-essential \
wget \
bzip2 \
ca-certificates \
libglib2.0-0 \
libxext6 \
libsm6 \
libxrender1 \
git
# Reduce the size of the image by deleting the package lists we downloaded,
# which are useless now.
apt-get -y clean
rm -rf /var/lib/apt/lists/*
#Installing Anaconda 3
wget -c https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
/bin/bash Anaconda3-2020.02-Linux-x86_64.sh -bfp /usr/local
#Conda configuration of channels from .condarc file
conda config --file /.condarc --add channels defaults
conda config --file /.condarc --add channels conda-forge
conda update conda
conda init bash
#List installed environments
conda list
Pytorch
BootStrap: docker
From: nvcr.io/nvidia/pytorch:24.05-py3
%post
# Downloads the latest package lists.
apt-get update -y
# Install Python3 requirements
# --> python3-tk is required by matplotlib.
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
python3 \
python3-tk \
python3-pip \
python3-distutils \
python3-setuptools
# Reduce the size of the image by deleting the package lists we downloaded,
# which are useless now.
apt-get -y clean
rm -rf /var/lib/apt/lists/*
# Install Python modules.
pip3 install numpy ipdb torch
Pytorch with ffcv
BootStrap: docker
From: nvcr.io/nvidia/pytorch:23.03-py3
%post
# Downloads the latest package lists.
apt-get update -y
export ARG DEBIAN_FRONTEND=noninteractive
export OPENCV_VERSION=4.7.0
apt-get update && apt-get upgrade -y &&\
# Install build tools, build dependencies and python
apt-get install -y \
python3-pip \
build-essential \
cmake \
git \
wget \
unzip \
yasm \
pkg-config \
libswscale-dev \
libtbb2 \
libtbb-dev \
libjpeg-dev \
libpng-dev \
libtiff-dev \
libavformat-dev \
libpq-dev \
libxine2-dev \
libglew-dev \
libtiff5-dev \
zlib1g-dev \
libjpeg-dev \
libavcodec-dev \
libavformat-dev \
libavutil-dev \
libpostproc-dev \
libswscale-dev \
libeigen3-dev \
libtbb-dev \
libgtk2.0-dev \
pkg-config \
## Python
python3-dev \
python3-numpy \
&& rm -rf /var/lib/apt/lists/*
cd /opt/ &&\
# Download and unzip OpenCV and opencv_contrib and delte zip files
wget https://github.com/opencv/opencv/archive/$OPENCV_VERSION.zip &&\
unzip $OPENCV_VERSION.zip &&\
rm $OPENCV_VERSION.zip &&\
wget https://github.com/opencv/opencv_contrib/archive/$OPENCV_VERSION.zip &&\
unzip ${OPENCV_VERSION}.zip &&\
rm ${OPENCV_VERSION}.zip &&\
# Create build folder and switch to it
mkdir /opt/build && cd /opt/build &&\
# Cmake configure
cmake \
-DOPENCV_EXTRA_MODULES_PATH=/opt/opencv_contrib-${OPENCV_VERSION}/modules \
-DWITH_CUDA=ON \
-DCUDA_ARCH_BIN=7.5,8.0,8.6 \
-DCMAKE_BUILD_TYPE=RELEASE \
-DOPENCV_GENERATE_PKGCONFIG=ON \
-DCMAKE_INSTALL_PREFIX=/usr/local ../opencv-${OPENCV_VERSION} \
&&\
# Make
make -j"$(nproc)" && \
# Install to /usr/local/lib
make install && \
ldconfig &&\
# Remove OpenCV sources and build folder
rm -rf /opt/opencv-${OPENCV_VERSION} && rm -rf /opt/opencv_contrib-${OPENCV_VERSION}
# install othere ffcv dependencies
pip3 install cupy-cuda113 numba
apt-get update -y
apt-get install -y libturbojpeg0-dev
# install ffcv
pip3 install ffcv
# add further packages below.
Install matlab into docker/singularity images
# --- MATLAB ---
ENV MATLAB_RELEASE=r2022a
# required packages depend a bit on which matlab toolboxes are required
RUN apt-get update \
&& apt-get install -yq --no-install-recommends \
libxt6 \
libx11-6 \
&& apt-get clean && rm -rf /var/lib/apt/lists/* \
&& locale-gen "en_US.UTF-8"
`# Run mpm to install MATLAB in the target location and delete the mpm installation afterwards.
`# If mpm fails to install successfully then output the logfile to the terminal, otherwise cleanup.
`# see https://de.mathworks.com/products.html for available products
RUN wget -q https://www.mathworks.com/mpm/glnxa64/mpm \
&& chmod +x mpm \
&& ./mpm install \
--release=${MATLAB_RELEASE} \
--destination=/opt/matlab \
--products MATLAB Deep_Learning_Toolbox Parallel_Computing_Toolbox Statistics_and_Machine_Learning_Toolbox Optimization_Toolbox Image_Processing_Toolbox \
|| (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) \
&& rm -f mpm /tmp/mathworks_root.log \
&& ln -s /opt/matlab/bin/matlab /usr/local/bin/matlab
ENV MLM_LICENSE_FILE=put_license_file
ENV MW_DDUX_FORCE_ENABLE=true MW_CONTEXT_TAGS=MATLAB:DOCKERFILE:V1
Note
The matlab-singularity example was kindly provided by Matthias Kümmerer.
Created: June 21, 2024