Myria calculator user guide¶

If you are new to the HPC environment, we recommend that you read the detailed version at format PDF. Otherwise, please read at least the first two paragraphs of the summary documentation before contacting support.

Connection, environment and workspaces¶

The connection is done in SSH to the frontals grouped under the name myria.criann.fr.

Command line syntax : ssh monlogin@myria.criann.fr (by replacing mylogin with your login). Linux and Mac environments natively integrate the SSH protocol through the terminal. If you are under Windows environment, we recommend you to use the software MobaXTerm which will bring you a complete working environment based on the ssh protocol (screen export, file transfer).

The first time you log in, you are prompted to change your password. Read carefully what is asked: (current) Password is the current password and not the new one you want...

Customizations (environment variables, aliases) are done with a ~/.bash_profile (not ~/.bashrc) to create.

The user has a personal workspace in /home.

By default a user disk quota is set to 50 GB in /home; the command mmlsquota gpfs1:home provides the quota and the space the user occupies in the /home partition.

We encourage you to compute in the temporary scratch folders (/dlocal/run/<jobid>) created by the Slurm batch tool for each job. A quota of 10 million files is applied to the /dlocal tree; the mmlsquota gpfs1:dlocal command provides the quota and the number of files you own in the /dlocal/run folder.

If you feel that these limits (quotas) are too restrictive for you, do not hesitate to open a ticket with the support. These limits can be increased upon justified request.

For more information about this workspace and recommended commands for managing your files, see this page.

Data transfers¶

For the transfer of significant amounts of data between Myria and the outside world, the myria-transfert.criann.fr server should be preferred because it is connected to the SYVIK network via a 40 Gbit/s interface.

This machine gives access to the same /home directory as the front ends. The connection identifiers on myria-transfert.criann.fr are the same as on myria.criann.fr.

Transfers between Myria and IDRIS resources are only allowed through myria-transfer.criann.fr.

Firewall¶

If your structure has a firewall limiting the outgoing flows of your site, here are the ports to open:

• SSH connections (port 22) to the 4 front-ends behind the name myria.criann.fr
  • IPv4 : 195.221.27.66 (myria-front1.criann.fr) to 195.221.27.69 (myria-front4.criann.fr)
  • IPv6 : 2001:660:7401:273::66 (myria-front1.criann.fr) to 2001:660:7401:273::69 (myria-front4.criann.fr)
• SSH connections (port 22) to the data transfer node
  • IPv4 : 195.221.27.70 (myria-transfert.criann.fr)
  • IPv6 : 2001:660:7401:273::70 (myria-transfert.criann.fr)

Concerning the remote visualization sessions, to know the IP and the ports to open, please contact the support by mail at support@criann.fr

Available software¶

Most of the submission script templates for the following software are available in the directory /soft/slurm/criann_modeles_scripts.

If not, contact support@criann.fr (same as for requesting new software).

Application and library environments are accessed by modules (see the module avail, module help commands and the general documentation on modules).

Commercial software_ (license to be taken by the user)

• CFD/Mechanics : ANSYS Fluent, StarCCM+, FINE/Marine, Hyperworks

Free software

• Fluid mechanics :
  • FDS 5.5.0 (incendies)
  • Telemac v7p1r1, v8p1r1, v8p2r0 and v8p3r1
  • Code_Saturne 5.0.7 and 6.1.1
  • OpenFOAM 2.3.0, 2.4.0, 3.0.1, 3.0.x, 4.x-version-4.0, 4.x-version-4.1, 5.x-version-5.0, 5.x-20171030, 6-version-6, 6-20190304, 7-version-7, 8-version-8, 10-version-10, 1606+, 1612+, 1706, 1712, 1806, 1812, 1906, 1912, 2006, 2012, 2106, 2112, 2206 and 2212
  • Foam-extend 3.2 and 4.0
  • Delft3d4 6.03.00.141173
• Mechanics of structures :
  • Castem 2016, 2017, 2019 and 2021
  • Code_Aster 11.6.0 serial, 13.4.0 serial, 14.2.0_testing ("stabilized development version") serial and parallel (MPI), 14.6.0 serial
  • Salome-Meca 2017.0.2
  • LMGC90/user_2019_rc1
• Chemistry :
  • GAMESS 2013 and 2016
  • Dalton 2018
  • Psi4
  • Cfour v1
  • Open Babel 3.0.0
• Molecular dynamics :
  • GROMACS 5.1.4, 2019.2, 2020.4, 2021.1 and 2022.1 (Plumed patch or not)
  • NAMD 2.9 (patch Plumed), 2.12, 2.13 (Plumed patch or not) and 2.14
  • DL_POLY_CLASSIC 1.9
  • LAMMPS 7Aug19 and 7Jan2022
  • OpenMolcas 19.11
  • Quantum Espresso 6.5
• Climate : WRF 3.7.1, 3.9, 3.9.1 and 4.3
• Ocean : Croco v1.1
• Geodynamics :
  • SPECFEM2D 2018
  • SPECFEM3D 2021
• Mathematics :
  • Octave 3.8.2 and 4.2.1
  • R 3.3.2, 3.5.1, 3.5.2, 4.0.2, 4.0.3, 4.0.5, 4.1.1 and 4.1.2
  • Scilab 5.5.2 and 6.0.1
• IA – Deep Learning : one page is dedicated to the installed software library

Software under academic license

• Quantum chemistry :
  • GAUSSIAN 03
  • Shrödinger JAGUAR 8.0 and 9.4
• Molecular dynamics : Shrödinger Desmond 2016.3

Software for visualization or pre/post-processing

• Gmsh 2.16.0 and 4.5.1
• Neper 3.0.1
• Triangle 1.6
• ParaView 4.0.1, 5.2.0, 5.4.1, 5.6.0, 5.7.0 and 5.9.0
• Salome 7.8.0 and 8.5.0
• Visit 2.12.1 and 2.12.3
• Blender 2.76b and 2.79b
• ffmpeg 4.1
• Helix-OS
• XCrysDen 1.6.2
• VMD 1.9.3

Compilation¶

Intel's MPI compiler and library environments, version 2017, are enabled by default (type module list).

MPI codes are compiled using the commands mpiifort (FORTRAN), mpiicc (C, 2 "i" in the name of this command), mpiicpc (C++).

The optimization option for the general-purpose (Broadwell) processor architecture is -xCORE-AVX2 (for compiling on a frontend, or batch in a non-knl partition, the -xHost option is equivalent).

The optimization option for the manycore Xeon Phi KNL processor architecture is -xMIC-AVX512.

The generation of an optimized executable for both Broadwell and KNL architectures is possible, with the -axCORE-AVX2,MIC-AVX512 option.

Examples of optimization option sets for Broadwell, in order of increasing aggressiveness

• -O2 -xCORE-AVX2
• -O3 -xCORE-AVX2 -fp-model precise
• -O3 -xCORE-AVX2

Example of an optimization option set for KNL

• -O3 -xMIC-AVX512

Example of debugging options

• -O2 -g -traceback -check all

Sample Makefiles are provided in /soft/makefiles.

The modules environment facilitates the compilation of applications linked to the available libraries.

Submission environment (Slurm)¶

The directory /soft/slurm/criann_modeles_scripts contains the generic submission scripts (sequential code, OpenMP, MPI, hybrid, KNL, GPU) or application specific.

Memory¶

It is advisable to use Slurm's #SBATCH --mem directive, specifying the memory demand per compute node (rather than memory per reserved core, --mem-per-cpu).

If a parallel computation needs more than 4300MB of memory per MPI task, the standard Myria compute nodes must be depopulated by adding the directive #SBATCH --ntasks-per-node in the submission script.

For example, the following settings require a total of 140 MPI jobs with, per standard compute node (server), 14 MPI jobs and 120000 MB of memory (since these servers have 28 cores each, half of the compute units will be used but all RAM will be available):

# Partition (submission class)
#SBATCH --partition 2tcourt

# MPI tasks number
#SBATCH --ntasks 140

# MPI tasks per compute node
#SBATCH --ntasks-per-node 14

# Maximum memory per compute node (MB)
#SBATCH --mem 120000

Commands¶

This table provides useful commands for submitting jobs.

Environment variables¶

The following utility variables (non-exhaustive list) can be used in the user commands (Shell) of a submission script.

Partitions (submission classes)¶

The partitions use the 368 Broadwell dual-socket nodes (10,304 cores), the 10 Xeon Phi KNL nodes (640 cores) and the SMP node (256 cores).

The number of compute nodes accessed by jobs in different partitions is given below as a guide. Administrators sometimes adjust the size of these "firing windows" slightly depending on the observed load. On the other hand, the per-computation limits generally remain constant. In any case, these limits are not modified without an announcement to the users' mailing list.

Note: The reproducibility of an application's performance on the cluster can be enhanced by applying the #SBATCH --exclusive directive in the submission script, which enforces the execution of the submitted job on a set of unshared nodes. This directive is advisable for multi-node MPI production jobs, recommended for performance testing.

For production or scientific development jobs, in the particular case of sequential applications, it is preferable not to apply this directive (so as not to monopolize a compute node).

The partition is specified by the submission script with the #SBATCH --partition directive (or in the command line: sbatch --partition 2tcourt script_submission.sl)

Debugging jobs

This partition contains all servers equipped with the Xeon Broadwell processor.

Work on standard architecture (Broadwell)

The partitions are defined according to the resource quantities requested.

(*) Weekend schedule starts on Friday evening at 5 pm and ends on Sunday evening at 8 pm. During this period, the total number of cores used by a user can reach 4200 cores, thus allowing a job in the 2tcourt partition up to 4200 cores.

Jobs on specific architectures

Partitions match the type of the targeted architecture.

The disk partition should be specified for write-intensive chemistry applications. See the submission file templates in /soft/slurm/criann_modeles_scripts :

• job_Gamess2013.sl
• job_Gamess2016.sl
• job_Jaguar-8.0.sl
• job_Jaguar-9.4.sl
• job_Gaussian03.sl

GPGPU¶

A description of how to submit GPU jobs is available.

One paragraph is for users who need to compile or run multi-GPU applications (for the CUDA aware MPI case) on the V100-SXM2 architecture from Myria.

KNL¶

Servers with a Xeon Phi KNL processor have several configurations (clustering and fast on-board memory modes).

The KNL documentation page describes these configurations and the Slurm guidelines for specifying them for jobs.

Remote visualization¶

A specific documentation for visualization jobs is available.

Management of signals sent by Slurm¶

A specific documentation for catching and processing signals sent by Slurm is available.

Supplements¶

The detailed documentation in PDF format (paragraphs "Bonnes pratiques de soumission" and "Exécution des travaux : aspects avancés") provides useful information:

• Ratio of resource consumption (e.g. memory) per job
• Multi-step jobs (e.g. pre-processing, solver and post-processing) and dependencies between jobs.
• Consideration of network topology for MPI communication performance
• Optimal placement of parallel processes in a job for performance (when using depopulated compute nodes)

Tracking hourly consumption¶

From the frontend nodes, the maconso command displays your consumption as well as that of the project. If you are a project manager or deputy manager maconso -u shows the details for all accounts.

This information is also available in JSON format in the file ~/.acct_myria.json.

The consumption is updated every night.