System Architecture
All Ferranti nodes run Rocky 8.8 and resource allocation is managed using Slurm (resource manager). Global storage is provided by a Weka file system. Inter-node communication uses a InfiniBand 400G non-blocking network.
The system is composed of 2 login nodes, 2 CPU-only nodes, and 15 H100 nodes, housed in 10 air-cooled racks.
Login Nodes
Ferranti's two login nodes have the following configuration:
| Feature | Login Node |
|---|---|
| CPUs: | 2 x Intel Xeon Gold 6430, 32 cores |
| RAM: | 1024GB DDR5-4800 ECC RAM |
| Local Storage: | 700GB NVMe |
CPU Nodes
The 2 CPU-only compute nodes have the following hardware:
| Feature | Specifications |
|---|---|
| CPUs: | 2 x AMD EPYC 9654(96 cores, 2.4 GHz, 384 MB L3 Cache) |
| RAM: | 2304GB DDR5-4800 |
| Local Storage: | 50TB |
| Theoretical Peak Performance: | ~ |
GPU Compute Nodes
| Feature | Specifications | Specifications |
|---|---|---|
| Total Nodes: | 5 | 10 |
| Accelerators: | 8 Nvidia H100 / node | 8 Nvidia H100 / node |
| Accelerator connect: | SXM5 | SXM5 |
| GPU Memory: | 80GB HBM3 / card (bandwidth: 3.35TB/s) | 80GB HBM3 / card (bandwidth: 3.35TB/s) |
| CPUs: | 96 cores: 2 x Intel Xeon 8468, 48 cores/die, 2.1 GHz | 192 cores: 2 x AMD Genoa 9654, 96 cores/die, 2.4 Ghz |
| RAM: | 2048GB DDR5-4800 | 2304GB DDR5-4800 |
| NVIDIA Tensor Cores: | 528 / card | 528 / card |
| FP32 Cores: | 16896 / card | 16896 / card |
| FP64 Cores: | 8448 / card | 8448 / card |
| Local Storage: | 24TB NVMe | 60TB NVMe |
| HCA: | 6x NVidia Mellanox ConnectX-7 NDR400 | 6x NVidia Mellanox ConnectX-7 NDR400 |
| Theoretical Peak Performance: | (FP16) 15832 TFLOPS ; (TF32) 7912 TFLOPS ; (FP32) 536 TFLOPS ; (FP64) 272 TFLOPS | (FP16) 15832 TFLOPS ; (TF32) 7912 TFLOPS ; (FP32) 536 TFLOPS ; (FP64) 272 TFLOPS |
Ferranti Interconnect
Ferranti has a fat tree InfiniBand interconnect topology with the following composition:
| Feature | Specifications |
|---|---|
| Number of core switches: | 4 |
| Number of edge switches: | 7 |
| Interconnect topology and type: | NDR InfiniBand Fat Tree, non-blocking |
| Blocking factor: | 1:1 |
Last update:
October 18, 2024
Created: June 21, 2024
Created: June 21, 2024