The command “module avail” shows all available modules:

torel@srl-login1:~$ module avail
-------------------------------------------------------------------------------- /cm/local/modulefiles ---------------------------------------------------------------------------------
cluster-tools/8.2  cmd  dot  freeipmi/1.6.2  gcc/8.2.0  gcc/64/4.0.0rc5  ipmitool/1.8.18  module-git  module-info  modules  null  openldap  shared  use.own  
 
-------------------------------------------------------------------------------- /cm/shared/modulefiles --------------------------------------------------------------------------------
cm-hwloc/1.11.11            default-environment                            intel/daal/64/2019/2.187  mpc/gcc/1.1.0(default)           openmpi/pgi/3.1.3             
cuda10.0/blas/10.0.130      gcc/10.0.0-20190609                            intel/gdb/64/2019/2.187   mpfr/gcc/4.0.2(default)          openmpi/pgi/2019              
cuda10.0/fft/10.0.130       gdb/8.2                                        intel/ipp/64/2019/2.187   mpich/ge/gcc/64/3.3-182-cm8.2    pgi-llvm/64/18.4              
cuda10.0/nsight/10.0.130    gmp/gcc/6.1.2(default)                         intel/mkl/64(default)     mpich/ge/intel/64/3.3-182-cm8.2  pgi/pgi-llvm                  
cuda10.0/profiler/10.0.130  hdf5_18/1.8.20                                 intel/mkl/64/2019/2.187   mvapich2/gcc/64/2.3              pgi/pgi-nollvm                
cuda10.0/toolkit/10.0.130   intel-cluster-runtime/ia32/2019.1(default)     intel/mpi/32/2019/2.187   mvapich2/psmgcc/64/2.3           pgi/PrgEnv-pgi/19.5(default)  
cuda10.1/blas/10.1.105      intel-cluster-runtime/intel64(default)         intel/mpi/64(default)     openblas/dynamic/0.2.20          slurm/18.08.4                 
cuda10.1/fft/10.1.105       intel-cluster-runtime/intel64/2019.1(default)  intel/mpi/64/2019/2.187   openblas/dynamic/0.3.7(default)  
cuda10.1/nsight/10.1.105    intel/compiler/32/2019/19.0.2                  intel/tbb/32/2019/2.187   openmpi/gcc/64/1.10.7            
cuda10.1/profiler/10.1.105  intel/compiler/64(default)                     intel/tbb/64(default)     openmpi/gcc/64/4.0.1             
cuda10.1/toolkit/10.1.105   intel/compiler/64/2019/19.0.2                  intel/tbb/64/2019/2.187   openmpi/intel/64/1.10.7          
torel@srl-login1:~$

Load the desired modules with the command “module load”:

marikkes@srl-login1:~$ module load mpfr/gcc/4.0.2 gmp/gcc/6.1.2 mpc/gcc/1.1.0 gcc/9.1.1

The command “module list” is used to see what modules are loaded:

marikkes@srl-login1:~$ module list
Currently Loaded Modulefiles:
 1) slurm/18.08.4(default)    4) mpc/gcc/1.1.0(default)  
 2) mpfr/gcc/4.0.2(default)   5) gcc/9.1.1(default)      
 3) gmp/gcc/6.1.2(default)

There are multiple architectures in this cluster. In order to source the correct environment, you may add to your .bashrc file something like the below.

torel@srl-login1:~$ tail -21 .bashrc 
 
if [ "`uname -m`" = "x86_64" ];
  then
  module load slurm
  module load gcc/10.0.0-20190609
  module load cuda10.1/blas/10.1.105
  module load cuda10.1/fft/10.1.105 
  module load cuda10.1/nsight/10.1.105 
  module load cuda10.1/profiler/10.1.105
  module load cuda10.1/toolkit/10.1.105
  export CUDA_HOME=/cm/shared/apps/cuda10.1/toolkit/10.1.105
  module load openmpi/gcc/64/1.10.7
  export OMPI_MCA_btl_openib_allow_ib=1
  export OMPI_MCA_btl_openib_if_include="mlx5_2:1"
 
# PGI compiler 19.10 License server 
#
# export LM_LICENSE_FILE=27000@172.16.16.96
# module load openmpi/3.1.2/2019 pgi/19.10
 
fi

Nodes n005-n008 is equipment with Dual Processor (DP) CN9980 processors. The CN9980 is Cavium's latest generation topbin Arm8.1 chip. More details can be found on WikiChip's page for CN9980.

Note that Cavium has been gobbled by Marvell.

Product homepage ThunderX2 CN99XX Product Brief ThunderX2 CN99XX Reference Platform

The mission of PRACE (Partnership for Advanced Computing in Europe) is to enable high-impact scientific discovery and engineering research and development across all disciplines to enhance European competitiveness for the benefit of society.

PRACE Best Practice Guide - ARM64

SLURM is a cluster management and job scheduling system for large and small Linux clusters. SLURM v18.08.4 is installed on the eX³, and the module is loaded by default.

Using the SLURM job scheduler

SLURM Quick Start User Guide

There are several Message Passing Interface implementations installed on the eX³:

• Open MPI: Available for GCC, Intel and PGI. https://www.open-mpi.org/
• MPICH: Available for GCC and Intel. https://www.mpich.org/
• MVAPICH2: Only available for GCC. http://mvapich.cse.ohio-state.edu/

There are three versions of GCC installed on the eX³:

• GCC v8.2.0
• GCC v9.1.1
• GCC v9.2.0
• GCC v10.0.0 (experimental)

GCC v8.2.0 is a local modulefile and is loaded by default. To load GCC v9.1.1 use the command “module load mpfr/gcc/4.0.2 gmp/gcc/6.1.2 mpc/gcc/1.1.0 gcc/9.1.1”, and to load GCC v10.0.0 use the command “module load mpfr/gcc/4.0.2 gmp/gcc/6.1.2 mpc/gcc/1.1.0 gcc/10.0.0-20190609”. These modules can not be loaded out of order. If only one GCC version is loaded, the simple command “gcc [filename]” will compile the file. However, if you wish to use a specific GCC version you need to specify the version in the command, for example: “gcc-8 [filename]”. The command for the Fortran compiler is “gfortran”.

The base optimisation flag system for GCC is O specifying the level of optimisation, with 0 to 3 and s, g and fast being the different choices. The flag to parallelize the code with OpenMP when using the GCC compiler is -fopenmp. More flags and suggested optimisation options for GCC here.

GCC Optimisation Options

There are two versions of the Intel C++ compiler installed: Intel32 and Intel64, both v19.0.4. The difference is that Intel32 compiles for 32-bit architecture, while Intel64 compiles for 64-bit architecture. All the nodes in the eX³ are 64-bit, which means both code compiled with Intel64 and with Intel32 can run on all nodes, however Intel64 will usually generate faster code. Simply use the “module load” command to load the desired version, as no prerequisites are required to load the Intel compilers. The C++ command to compile the program is “icc [filename]”, while the Fortran command is “ifort [filename]”.

The Intel compiler has a similar base optimisation flag system to GCC, with O specifying the code optimisation for applications. The possible levels are 0, 1, 2 and 3. If not specified the default is O2. The flag to parallelize the code with OpenMP when using the Intel compiler is -qopenmp.

Alphabetical list of Intel C++ Compiler Options

There are two different variants of the PGI C++ compiler installed: the normal PGI compiler and the LLVM-based PGI compiler. For the normal compiler load “pgi/64/19.5” and for the LLVM one load “pgi-llvm/64/19.5”. For C++, the command for the normal PGI is “pgcc [filename]” while the command for the LLVM-based PGI is “pgcc-llvm [filename]”. For Fortran, the command for the normal PGI is “pgfortran [filename]” while the command for the LLVM-based PGI is “pgfortran-llvm [filename]”. From benchmarking with McCalpin STREAM, LLVM generally returns a higher performance than the regular PGI compiler.

The base optimisation flag system has 5 levels, from -O0 to -O4, similar to the Intel and GCC compilers. The flag to parallelize the code with OpenMP is -mp for both PGI Compilers.

PGI Compiler User´s Guide

PGI Compiler Reference Manual