The command “module avail” shows all available modules:
torel@srl-login1:~$ module avail -------------------------------------------------------------------------------- /cm/local/modulefiles --------------------------------------------------------------------------------- cluster-tools/8.2 cmd dot freeipmi/1.6.2 gcc/8.2.0 gcc/64/4.0.0rc5 ipmitool/1.8.18 module-git module-info modules null openldap shared use.own -------------------------------------------------------------------------------- /cm/shared/modulefiles -------------------------------------------------------------------------------- cm-hwloc/1.11.11 default-environment intel/daal/64/2019/2.187 mpc/gcc/1.1.0(default) openmpi/pgi/3.1.3 cuda10.0/blas/10.0.130 gcc/10.0.0-20190609 intel/gdb/64/2019/2.187 mpfr/gcc/4.0.2(default) openmpi/pgi/2019 cuda10.0/fft/10.0.130 gdb/8.2 intel/ipp/64/2019/2.187 mpich/ge/gcc/64/3.3-182-cm8.2 pgi-llvm/64/18.4 cuda10.0/nsight/10.0.130 gmp/gcc/6.1.2(default) intel/mkl/64(default) mpich/ge/intel/64/3.3-182-cm8.2 pgi/pgi-llvm cuda10.0/profiler/10.0.130 hdf5_18/1.8.20 intel/mkl/64/2019/2.187 mvapich2/gcc/64/2.3 pgi/pgi-nollvm cuda10.0/toolkit/10.0.130 intel-cluster-runtime/ia32/2019.1(default) intel/mpi/32/2019/2.187 mvapich2/psmgcc/64/2.3 pgi/PrgEnv-pgi/19.5(default) cuda10.1/blas/10.1.105 intel-cluster-runtime/intel64(default) intel/mpi/64(default) openblas/dynamic/0.2.20 slurm/18.08.4 cuda10.1/fft/10.1.105 intel-cluster-runtime/intel64/2019.1(default) intel/mpi/64/2019/2.187 openblas/dynamic/0.3.7(default) cuda10.1/nsight/10.1.105 intel/compiler/32/2019/19.0.2 intel/tbb/32/2019/2.187 openmpi/gcc/64/1.10.7 cuda10.1/profiler/10.1.105 intel/compiler/64(default) intel/tbb/64(default) openmpi/gcc/64/4.0.1 cuda10.1/toolkit/10.1.105 intel/compiler/64/2019/19.0.2 intel/tbb/64/2019/2.187 openmpi/intel/64/1.10.7 torel@srl-login1:~$
Load the desired modules with the command “module load”:
marikkes@srl-login1:~$ module load mpfr/gcc/4.0.2 gmp/gcc/6.1.2 mpc/gcc/1.1.0 gcc/9.1.1
The command “module list” is used to see what modules are loaded:
marikkes@srl-login1:~$ module list Currently Loaded Modulefiles: 1) slurm/18.08.4(default) 4) mpc/gcc/1.1.0(default) 2) mpfr/gcc/4.0.2(default) 5) gcc/9.1.1(default) 3) gmp/gcc/6.1.2(default)
There are multiple architectures in this cluster. In order to source the correct environment, you may add to your .bashrc file something like the below.
torel@srl-login1:~$ tail -21 .bashrc if [ "`uname -m`" = "x86_64" ]; then module load slurm module load gcc/10.0.0-20190609 module load cuda10.1/blas/10.1.105 module load cuda10.1/fft/10.1.105 module load cuda10.1/nsight/10.1.105 module load cuda10.1/profiler/10.1.105 module load cuda10.1/toolkit/10.1.105 export CUDA_HOME=/cm/shared/apps/cuda10.1/toolkit/10.1.105 module load openmpi/gcc/64/1.10.7 export OMPI_MCA_btl_openib_allow_ib=1 export OMPI_MCA_btl_openib_if_include="mlx5_2:1" # PGI compiler 19.10 License server # # export LM_LICENSE_FILE=27000@172.16.16.96 # module load openmpi/3.1.2/2019 pgi/19.10 fi
Nodes n005-n008 is equipment with Dual Processor (DP) CN9980 processors. The CN9980 is Cavium's latest generation topbin Arm8.1 chip. More details can be found on WikiChip's page for CN9980.
Note that Cavium has been gobbled by Marvell.
The mission of PRACE (Partnership for Advanced Computing in Europe) is to enable high-impact scientific discovery and engineering research and development across all disciplines to enhance European competitiveness for the benefit of society.
SLURM is a cluster management and job scheduling system for large and small Linux clusters. SLURM v18.08.4 is installed on the eX³, and the module is loaded by default.
There are several Message Passing Interface implementations installed on the eX³:
There are three versions of GCC installed on the eX³:
GCC v8.2.0 is a local modulefile and is loaded by default. To load GCC v9.1.1 use the command “module load mpfr/gcc/4.0.2 gmp/gcc/6.1.2 mpc/gcc/1.1.0 gcc/9.1.1”, and to load GCC v10.0.0 use the command “module load mpfr/gcc/4.0.2 gmp/gcc/6.1.2 mpc/gcc/1.1.0 gcc/10.0.0-20190609”. These modules can not be loaded out of order. If only one GCC version is loaded, the simple command “gcc [filename]” will compile the file. However, if you wish to use a specific GCC version you need to specify the version in the command, for example: “gcc-8 [filename]”. The command for the Fortran compiler is “gfortran”.
The base optimisation flag system for GCC is O specifying the level of optimisation, with 0 to 3 and s, g and fast being the different choices. The flag to parallelize the code with OpenMP when using the GCC compiler is -fopenmp. More flags and suggested optimisation options for GCC here.
There are two versions of the Intel C++ compiler installed: Intel32 and Intel64, both v19.0.4. The difference is that Intel32 compiles for 32-bit architecture, while Intel64 compiles for 64-bit architecture. All the nodes in the eX³ are 64-bit, which means both code compiled with Intel64 and with Intel32 can run on all nodes, however Intel64 will usually generate faster code. Simply use the “module load” command to load the desired version, as no prerequisites are required to load the Intel compilers. The C++ command to compile the program is “icc [filename]”, while the Fortran command is “ifort [filename]”.
The Intel compiler has a similar base optimisation flag system to GCC, with O specifying the code optimisation for applications. The possible levels are 0, 1, 2 and 3. If not specified the default is O2. The flag to parallelize the code with OpenMP when using the Intel compiler is -qopenmp.
There are two different variants of the PGI C++ compiler installed: the normal PGI compiler and the LLVM-based PGI compiler. For the normal compiler load “pgi/64/19.5” and for the LLVM one load “pgi-llvm/64/19.5”. For C++, the command for the normal PGI is “pgcc [filename]” while the command for the LLVM-based PGI is “pgcc-llvm [filename]”. For Fortran, the command for the normal PGI is “pgfortran [filename]” while the command for the LLVM-based PGI is “pgfortran-llvm [filename]”. From benchmarking with McCalpin STREAM, LLVM generally returns a higher performance than the regular PGI compiler.
The base optimisation flag system has 5 levels, from -O0 to -O4, similar to the Intel and GCC compilers. The flag to parallelize the code with OpenMP is -mp for both PGI Compilers.