Skip to content

Singularity Containers

To ensure your results are reproducible and verifiable, use containers to package your research projects. Containers include an entire operating system, GPU libraries, etc., so this page discusses what a container is and how to work with our Singularity container system.

Docker images are not secure because they provide a means to gain root access to the system they are running on. This is not a problem because you can use Singularity which is an alternative to Docker that is both secure and designed for high-performance computing. Singularity is compatible with all Docker images and it can be used with GPUs and MPI applications, as well as Infiniband networks.

Singularity is already installed globally on all our systems, and should be immediately available on your command line. You can check the singularity version:

singularity --version
If you want to use Docker and other format containers they need to be converted to singularity format for use on the cluster.

GPU support for containers

Commands that run, or otherwise execute (singularity) containers (shell, exec) can take an --nv option, which will setup the container’s environment to use an NVIDIA GPU and the basic CUDA libraries to run a CUDA enabled application. The --nv flag will:

  • Ensure that the /dev/nvidiaX device entries are available inside the container, so that the GPU cards in the host are accessible.
  • Locate and bind the basic CUDA libraries from the host into the container, so that they are available to the container, and match the kernel GPU driver on the host.
  • Set the LD_LIBRARY_PATH inside the container so that the bound-in version of the CUDA libraries are used by applications run inside the container.

Converting Docker Image to Singularity

If you want to convert a docker image called ubuntu with the tag latest to a singularity file called IMAGE_NAME.sif, just run this command:

singularity build IMAGE_NAME.sif docker://ubuntu:latest

You can execute this command on the login nodes, as well as on any of the compute nodes of the clusters. Also, you can find more information here.

Use docker2singularity

If you have access to a computer with docker installed (and running) you can also use the singularity2docker tool for converting your images. To convert the ubuntu image to singularity (as above) use:

docker run -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp/test:/output \
  --privileged -t --rm \
  quay.io/singularity/docker2singularity \
  ubuntu:latest
This will create a *.simg file you can then transfer to a ML-Cloud server. By itself this is more complicated then the first method, but it can become necessary if you want to make adjustements to a container that are difficult to reproduce in a singularity recipe.

You can find more about singularity2docker usage on github.

Build Singularity on ML Cloud Example

You can build a customer image by using one of the published images as base image.

First, Set the cache and tmp directories for singularity. This is required to build the container, as otherwise storage can be insufficient and result in a build failure.

export SINGULARITY_CACHEDIR=#SCRATCH/group-name/your-username
export SINGULARITY_TMPDIR=#SCRATCH/group-name/your-username

Mounting behavior

Singularity by default mounts these 5 folders from the host filesystem into the container filesystem:

/home/$USER
/tmp
/dev
/sys
/proc

so if your script or data is in one of these folders then you don't need manual mounting and can simply specify the path to the files to singularity commands.

Creating a definition file

To build an image , you need to create a file called definition file. This is a file with some instructions telling Singularity which image should be used as base image and which commands you would like to run. Then execute:

singularity build --fakeroot your-custom-image.sif def-file-example.de

where def-file-example.def if the definition file. To build an image without root privileges, use --fakeroot.

See Example Definition Files

Cuda with Conda

BootStrap: docker
From: nvcr.io/nvidia/cuda:12.5.0-devel-ubuntu22.04
%post
    # Downloads the latest package lists.
    apt-get update -y

    # Install Python3 requirements
    # --> python3-tk is required by matplotlib.
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        python3 \
        python3-tk \
        python3-pip \
        python3-distutils \
        python3-setuptools \
    build-essential \
    wget \
    bzip2 \
    ca-certificates \
    libglib2.0-0 \
    libxext6 \
    libsm6 \
    libxrender1 \
    git

    # Reduce the size of the image by deleting the package lists we downloaded,
    # which are useless now.
    apt-get -y clean
    rm -rf /var/lib/apt/lists/*

    #Installing Anaconda 3 
    wget -c https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
    /bin/bash Anaconda3-2020.02-Linux-x86_64.sh -bfp /usr/local
    #Conda configuration of channels from .condarc file
    conda config --file /.condarc --add channels defaults
    conda config --file /.condarc --add channels conda-forge
    conda update conda
    #List installed environments
    conda list

Pytorch with conda

BootStrap: docker
From: nvcr.io/nvidia/pytorch:24.05-py3

%post

    # Downloads the latest package lists.
    apt-get update -y

    # Install Python3 requirements
    # --> python3-tk is required by matplotlib.
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        python3 \
        python3-tk \
        python3-pip \
        python3-distutils \
        python3-setuptools \
    build-essential \
    wget \
    bzip2 \
    ca-certificates \
    libglib2.0-0 \
    libxext6 \
    libsm6 \
    libxrender1 \
    git

    # Reduce the size of the image by deleting the package lists we downloaded,
    # which are useless now.
    apt-get -y clean
    rm -rf /var/lib/apt/lists/*

    #Installing Anaconda 3 
    wget -c https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
    /bin/bash Anaconda3-2020.02-Linux-x86_64.sh -bfp /usr/local
    #Conda configuration of channels from .condarc file
    conda config --file /.condarc --add channels defaults
    conda config --file /.condarc --add channels conda-forge
    conda update conda
    conda init bash
    #List installed environments
    conda list

Pytorch

BootStrap: docker
From: nvcr.io/nvidia/pytorch:24.05-py3
%post
    # Downloads the latest package lists.
    apt-get update -y

    # Install Python3 requirements
    # --> python3-tk is required by matplotlib.
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
        python3 \
        python3-tk \
        python3-pip \
        python3-distutils \
        python3-setuptools

    # Reduce the size of the image by deleting the package lists we downloaded,
    # which are useless now.
    apt-get -y clean
    rm -rf /var/lib/apt/lists/*

    # Install Python modules.
    pip3 install numpy ipdb torch

Pytorch with ffcv

BootStrap: docker
From: nvcr.io/nvidia/pytorch:23.03-py3
%post
    # Downloads the latest package lists.
    apt-get update -y

    export ARG DEBIAN_FRONTEND=noninteractive
    export  OPENCV_VERSION=4.7.0
apt-get update && apt-get upgrade -y &&\
    # Install build tools, build dependencies and python
    apt-get install -y \
    python3-pip \
        build-essential \
        cmake \
        git \
        wget \
        unzip \
        yasm \
        pkg-config \
        libswscale-dev \
        libtbb2 \
        libtbb-dev \
        libjpeg-dev \
        libpng-dev \
        libtiff-dev \
        libavformat-dev \
        libpq-dev \
        libxine2-dev \
        libglew-dev \
        libtiff5-dev \
        zlib1g-dev \
        libjpeg-dev \
        libavcodec-dev \
        libavformat-dev \
        libavutil-dev \
        libpostproc-dev \
        libswscale-dev \
        libeigen3-dev \
        libtbb-dev \
        libgtk2.0-dev \
        pkg-config \
        ## Python
        python3-dev \
        python3-numpy \
    && rm -rf /var/lib/apt/lists/*


cd /opt/ &&\
    # Download and unzip OpenCV and opencv_contrib and delte zip files
    wget https://github.com/opencv/opencv/archive/$OPENCV_VERSION.zip &&\
    unzip $OPENCV_VERSION.zip &&\
    rm $OPENCV_VERSION.zip &&\
    wget https://github.com/opencv/opencv_contrib/archive/$OPENCV_VERSION.zip &&\
    unzip ${OPENCV_VERSION}.zip &&\
    rm ${OPENCV_VERSION}.zip &&\
    # Create build folder and switch to it
    mkdir /opt/build && cd /opt/build &&\
    # Cmake configure
    cmake \
        -DOPENCV_EXTRA_MODULES_PATH=/opt/opencv_contrib-${OPENCV_VERSION}/modules \
        -DWITH_CUDA=ON \
        -DCUDA_ARCH_BIN=7.5,8.0,8.6 \
        -DCMAKE_BUILD_TYPE=RELEASE \
    -DOPENCV_GENERATE_PKGCONFIG=ON \
        -DCMAKE_INSTALL_PREFIX=/usr/local ../opencv-${OPENCV_VERSION} \
         &&\
    # Make
    make -j"$(nproc)" && \
    # Install to /usr/local/lib
    make install && \
    ldconfig &&\
    # Remove OpenCV sources and build folder
    rm -rf /opt/opencv-${OPENCV_VERSION} && rm -rf /opt/opencv_contrib-${OPENCV_VERSION}


# install othere ffcv dependencies
  pip3 install cupy-cuda113 numba


  apt-get update -y
  apt-get install -y libturbojpeg0-dev 

# install ffcv
  pip3 install ffcv

# add further packages below.

Install matlab into docker/singularity images

 # --- MATLAB ---
ENV MATLAB_RELEASE=r2022a

# required packages depend a bit on which matlab toolboxes are required
RUN apt-get update \
 && apt-get install -yq --no-install-recommends \
   libxt6 \
   libx11-6 \
 && apt-get clean && rm -rf /var/lib/apt/lists/* \
 && locale-gen "en_US.UTF-8"

`# Run mpm to install MATLAB in the target location and delete the mpm installation afterwards.
`# If mpm fails to install successfully then output the logfile to the terminal, otherwise cleanup.
`# see https://de.mathworks.com/products.html for available products
RUN wget -q https://www.mathworks.com/mpm/glnxa64/mpm \
    && chmod +x mpm \
    && ./mpm install \
    --release=${MATLAB_RELEASE} \
    --destination=/opt/matlab \
    --products MATLAB Deep_Learning_Toolbox Parallel_Computing_Toolbox Statistics_and_Machine_Learning_Toolbox Optimization_Toolbox Image_Processing_Toolbox \ 
    || (echo "MPM Installation Failure. See below for more information:" && cat /tmp/mathworks_root.log && false) \
    && rm -f mpm /tmp/mathworks_root.log \
    && ln -s /opt/matlab/bin/matlab /usr/local/bin/matlab

ENV MLM_LICENSE_FILE=put_license_file
ENV MW_DDUX_FORCE_ENABLE=true MW_CONTEXT_TAGS=MATLAB:DOCKERFILE:V1

Note

The matlab-singularity example was kindly provided by Matthias Kümmerer.

Last update: July 3, 2024
Created: July 3, 2024