Simula eX³ cluster wiki

The AMD EPYC 7601 is a x86-64 microprocessor. It is based on AMD´s Zen microarchitecture and is specifically targeted for server and embedded system markets. There are four dual processor AMD nodes in the eX³.

• Packages (sockets): 2
• Cores: 64
• Processors (CPUs): 128
• Cores per package: 32
• Threads per core: 2
• Base clock: 2.2GHz
• Max boost clock: 3.2GHz
• Partition: defq
• Nodes: n[001-004]

More information:

• AMD Epyc 7601 Official Page
• AMD Epyc Wikipedia Page
• PRACE Best Practice Guide AMD Epyc

Cavium ThunderX2 CN9980 is a 64-bit multi-core ARM server microprocessor. ARM processors are heavily used in mobile phones and have the reputation of being very energy efficient. The ARM nodes are currently not up.

• Number of CPU cores: 64
• Number of threads: 128
• Base clock: 2.2GHz
• Max boost clock: 2.5GHz
• Partition: armq
• Nodes: n[005-008] Note! n008 is currently lend to Dolphin Ics

More information:

• Cavium ThunderX2 CN9980 Wikichip Page
• Cavium ThunderX2 Anandtech Assessment
• PRACE Best Practice Guide ARM64

Intel Xeon Platinum 8168 is a 64-bit 24-core x86 multi-socket highest performance server microprocessor. There is one dual processor Xeon Platinum node in the eX³.

• Packages (sockets): 2
• Cores: 48
• Processors (CPUs): 96
• Cores per package: 24
• Threads per core: 2
• Base clock: 2.70GHz
• Max boost clock: 3.70GHz
• Partition: dgx2q
• Nodes: g001

More information:

• Intel Xeon Platinum 8168 Official Page
• Intel Xeon Platinum 8168 Wikichip Page

Intel Xeon Gold 6130 is a 64-bit 16-core x86 multi-socket high performance server microprocessor. There are four Xeon Gold nodes in the eX³ and they function as the log in-nodes.

• Packages (sockets): 2
• Cores: 32
• Processors (CPUs): 64
• Cores per package: 16
• Threads per core: 2
• Base clock: 2.1GHz
• Max boost clock: 3.7GHz
• Partition: xeongold16q
• Nodes: srl-[login1, adm2, mds1, mds2]

More information:

• Intel Xeon Gold 6130 Official Page
• Intel Xeon Gold 6130 Wikichip Page

Intel Xeon Silver 4112 is a 64-bit quad-core x86 dual-socket mid-range performance server microprocessor. There are eight single processor Xeon Silver nodes in the eX³.

• Packages (sockets): 1
• Cores: 4
• Processors (CPUs): 8
• Cores per package: 4
• Threads per core: 2
• Base clock: 2.6GHz
• Max boost clock: 3GHz
• Partition: slowq
• Nodes: n[041-048]

More information:

• Intel Xeon Silver 4112 Official Page
• Intel Xeon Silver 4112 Wikichip Page

To get more information on a specific node, run the file “cpuinfo” in “/cm/shared/apps/intel/compilers_and_libraries/2019.4.243/linux/mpi/intel64/bin/” while logged in to the node in question.

Two nodes on eX³ has special Non-Volatile Memory Dual-Inline-Memory-Modules (DIMMs).

srl-mds1 and srl-mds2 are the Metadata servers for BeeGFS filesystem. NVDIMMs will eventually be used to cache BeeGFS MetaData traffic, while being buddy-mirrored to spinning disk on a NetApp E5760i storage system.

To get access to the systems to test PMEM, PMDK development etc. please contact Tore H. Larsen

These pages contain instructions, links and other information related to persistent memory in Linux.

• Linux Kernel NVDIMM Wiki: https://nvdimm.wiki.kernel.org/start
• PMEM.IO homepage: http://pmem.io/
• NVDIMM Namespace: http://pmem.io/documents/NVDIMM_Namespace_Spec.pdf
• Driver Writer’s Guide: http://pmem.io/documents/NVDIMM_Driver_Writers_Guide.pdf
• NVDIMM Kernel Tree: https://git.kernel.org/cgit/linux/kernel/git/nvdimm/nvdimm.git
• NDCTL: https://github.com/pmem/ndctl.git
• NDCTL man pages online: http://pmem.io/ndctl/
• linux-nvdimm Mailing List: https://lists.01.org/mailman/listinfo/linux-nvdimm
• linux-nvdimm Patchwork: https://patchwork.kernel.org/project/linux-nvdimm/list/

• Medium article on Enabling-NVDIMMs-on-Ubuntu
• NVDIMM Enabling in SUSE Linux Enterprise 12, Service Pack 2 - Part 1
• NVDIMM Enabling in SUSE Linux Enterprise 12, Service Pack 2 - Part 2

• Advanced Configuration and Power Interface (ACPI) 6.2a: http://www.uefi.org/sites/default/files/resources/ACPI%206_2_A_Sept29.pdf
• Unified Extensible Firmware Interface (UEFI) Specification 2.7: http://www.uefi.org/sites/default/files/resources/UEFI_Spec_2_7.pdf
• DMTF System Management BIOS (SMBIOS 3.2.0): https://www.dmtf.org/sites/default/files/standards/documents/DSP0134_3.2.0.pdf
• JEDEC Byte Addressable Energy Backed Interface (JESD245B.01): https://www.jedec.org/system/files/docs/JESD245B-01.pdf
• JEDEC DDR4 NVDIMM-N Design Specification (JESD248A): https://www.jedec.org/system/files/docs/JESD248A.pdf

• Intel Optane DC persistent memory management software: https://github.com/intel/ipmctl
• Intel DSM Interface: http://pmem.io/documents/NVDIMM_DSM_Interface-V1.7.pdf
• Microsoft DSM Interface: https://docs.microsoft.com/en-us/windows-hardware/drivers/storage/-dsm-interface-for-byte-addressable-energy-backed-function-class--function-interface-1-
• HPE DSM Interface: https://github.com/HewlettPackard/hpe-nvm