Getting started

This guide is dedicated to the usage of MeSU-alpha and MeSU-beta computing servers, and should help you to get a better understanding of the machines you are going to use, and provide you with enough elements to run your first jobs. Make sure to fully understand the first sections content, in order to be able to follow the walkthrough in the “try it yourself” section.

A note on parallel computing: Newcomers unfamiliar with the basis of parallel computing are invited to consult the excellent LLNL tutorial on the subject, as well as an explanation of the differences between the two main programming methods used in parallel computing: OpenMP & MPI.

1 – System Overview

 

The global HPCaVe servers architecture is more complex than a personal computer. They are made of two kinds of nodes (a node represents a subset of a server, dedicated to specific tasks) – login nodes and computing nodes – which communicate together via a specific program called a batch scheduler.

 

1.1 – Login nodes

In order to run a program on one of our two computing servers (MeSU alpha and MeSU beta), you will have to first log into the login nodes. Login nodes are dedicated and separate small servers which act as a secure gateway between your computer and the computing nodes, on which your program will be running.

Note: login nodes are not meant for heavy computation, and any intensive job running on them will be automatically killed.

The login nodes are dedicated to the following tasks:

  • Transferring files from/to your personal computer.
  • Compiling your source code
  • Setting up your jobs’ parameters and variables
  • Monitoring the execution of your jobs

 

1.2 – Computing nodes (on MeSU alpha and MeSU beta)

Computing nodes constitute the core of the computing system. At HPCaVe, the computing nodes are distributed on two machines: MeSU alpha and MeSU beta.

Your jobs will run on those nodes, and will be submitted, managed and monitored with the batch scheduler.

 

1.3 – Data directories

Different filesystems are mounted on MeSU servers. When logged in, you have a direct access to your /home directory, which is meant to host your codes, programs, and allow you to submit jobs. This directory is not meant for data storage, and is limited to 30Gb per user.
Other directories are to be used for data storage:

  • /scratchalpha/username: Data directory, optimized for MeSU-alpha usage (250 Gb per user)
  • /scratchbeta/username: Data directory, optimized for MeSU-beta usage (500 Gb per user)

Note: data located on /scratchalpha is only accessible from MeSU-alpha, and data on /scratchbeta from MeSU-beta.

1.4 – The batch scheduler

A batch scheduler is a very specific software managing all users’ jobs submissions, in order to prioritize and allocate resources for every of them. Acting as a gateway between the login nodes and the computing nodes, its role is to optimize the cluster utilization according to the stack of pending jobs, and their requested resources (execution time, number of CPUs, maximum memory…). The batch scheduler used on HPCaVe servers is called PBS.

https://software.ecmwf.int/wiki/download/attachments/47294624/batch-schema.png?version=1&modificationDate=1430408545358&api=v2

2 – Connection to HPCave servers

 

If you did not already opened an account, please consult the dedicated page.

In order to launch your first jobs, you will first need to login to the login nodes, using the Secure Shell communication protocol (ssh), which guarantees a secure transfer of data.

You can either use ssh via a command line interface (CLI) or a graphical application.

To connect, type in the terminal (make sure to replace “your-username” with your real username) :

or use the X11 session forwarding if you want to have access to the GUI (Graphical User Interface) of the softwares you are using:

After typing in your password, you should be welcomed with a message provided by the system administrator, which will give you information about the current status of MeSU and future maintenance operations:

 

 

 

3 – Uploading files from your computer (and back!)

 

In order to upload files to MeSU servers, you will either need to use a specialized FTP software such as Filezilla, or use terminal UNIX commands such as scp, rsync, fputs

Please note that the servers are connected to internet, and you can therefore also download items with the command wget for instance, or by checking out a git, svn or mercurial repository:

 

3.1 – Filezilla (Windows, MacOsX and linux)

 

Filezilla is one of the most popular FTP (File Transfer Protocol) software available, and allows for an easy transfer of data between two network machines.

On the following screenshot, please fill the fields of the zone 2 as follows:

  • Host: sftp://mesu.dsi.upmc.fr
  • Username: Your HPCaVe username
  • Password
  • Port: 22 is the standard port for ssh protocol

Once you are done, hit the Quickconnect button, and transfer files between the zone 4 (your local computer) and the zone 5 (HPCaVe servers).

 

 

3.2 – scp (linux & MacOsX)

 

On a linux or MacOSX system (and through MobaXTerm for instance for Windows users), to transfer a source file called main.cpp to your session on HPCaVe servers:

The scp command also supports wildcards (*) selection and directory copying:

In order to copy a file back from HPCaVe servers to your personal computer, you will first have to retrieve your local IP adress:

  • “ifconfig” in a terminal in linux
  • Apple Menu > System preferences > Network on MacOSX

then, while logged in to HPCaVe login nodes, type in a terminal (“user” is the username on your personal computer):

 

 

4 – Running a job

 

You will mainly use HPCaVe servers to run two very distinct types of programs:

  1. Already installed softwares.
  2. Softwares of which you have the source code (on your computer, or through a remote source control system), and need to install (compile).

Although the main linux utilities and compilers (grep, cat, gcc…) are available in your environment as soon as you connect to HPCaVe servers, not all functionalities and software are accessible by default.

If you wish to use a specific scientific software (Matlab, gromacs, freefem++ …) or a certain suite of developing tools (mkl, intel Vtunes, specific compiler versions…), you will have to use modules.

 

4.1 – A note on “modules” (working with an installed software)

Modules act as software packages which you can load interactively in order to gain access to their contents. They are used to avoid conflicts between software and libraries versions, and to provide you at all time with a clean working environment ( for instance no environment variables, defining the location of software on the servers, should have the same name but point to different paths).

In order to know which modules are available and see if you need to use them, you can either consult the dedicated page or type the command:

To load a module and gain access to its content, type:

The specific program or software should now be available at the command line.

 

4.2 – Compiling your code

If you have access to the source code of an external software, or if you developed your own code, you will most probably be able to compile and run it on HPCaVe servers.

Compiling a code (transforming the source code in a binary executable file) is a whole topic by itself, but many tools are available on HPCaVe servers in regards with code development, for instance:

  • Collection of open source compilers (gcc, gfortan, mpifortran, mpic++, mpicc…) in recent and older versions
  • Extensive collection of scientific libraries
  • Optimized versions of MPI and OpenMP for the HPCaVe servers
  • Intel compilers and scientific libraries such as mkl
  • Profiling and debugging tools (gdb, gprof, valgrind, Intel VTunes…)
  • Text editors (vi, vim, GNU Emacs, nano)

Note: If you did not load the correct modules before compiling your code, your compiler should output error messages specifying the missing libraries.

 

4.3 – Set up your job script from /HOME

As explained in the Overview section, the communication between the login nodes and the compute nodes is managed by PBS, to which you will request computing resources, specify your job parameters and commands to execute through a shell or python file, called a PBS script.

Whether you wish to work with an existing software or with a compiled program, you will therefore have to write such a file.

Minimal script example

Explanation

Note that although the line starting with a “#” will be parsed as comments, the first few lines of the file will allow you to specify different resources.

For instance, to request 32 cores on MeSU alpha, you should replace the line 3 with:

To request 96 cores distributed on MeSU beta, you should actually request to use 4 nodes (see MeSU-beta technical specifications) containing each 24 cores:

If your job should run for 16 hours (more than the 8 hours allowed by default), add the line:

 

NB: Be aware that your program will run in a temporary directory on the compute nodes, which won’t be accessible anymore at the end of the execution. You must therefore transfer any output data back to your working environment ($PBS_O_WORKDIR) on the login nodes (line 11).

from scratch

 

4.4 – Set up your job script from /scratch directories

The scratch directories are created on your first login, and are located at /scratchalpha/$USER and /scratchbeta/$USER

If you do not tell PBS to use those spaces explicitly, your job will be executed from your current working directory (most often your home directory).

Steps to use scratch directories
  • Compile your code if needed (you can do it in your /home directory, but also in your scratch spaces).
  • Move any input data files to the appropriate scratch directory. If you plan on using MeSU-alpha for instance, move your input files to /scratchalpha/$USER
  • If you wish to run your job directly from /scratchalpha:
    • Copy your executable and your PBS script in a dedicated directory in your scratch space.
    • Change your working directory to /scratchalpha/$USER with cd
    • Launch your script with qsub from /scratchalpha
  • If you wish to run your job from your /home directory:
    • edit your PBS script to use the appropriate environment variables and read/write from/to the correct directories
    • Launch your script from your home directory
Script examples

For instance, here is a script you could launch from your /home directory, which will write in your /scratchalpha directory:

And here is a sample file for MeSU-beta:

 

4.5 – Submit a job

Once the PBS script has been correctly set up, requesting your job to run on the compute nodes is then as simple as executing the qsub command:

which will output on success a unique string, the job identifier (referred to later as jobID), in the form “123456.mesu1” which will be useful to get information on your job status.

If not specified, PBS will try to send your job to MeSU-alpha or MeSU-beta depending on the resources requested. If you know which server you want to run on, you can submit your job with one of the following commands:

 

 

5 – Checking your job status

 

The jobs you will submit to PBS do not usually run as soon as you have ran the command qsub. Indeed, PBS first has to parse your script file in order to insert your job in the queue system. Depending on the resources (number of nodes and CPU cores, maximum memory, walltime i.e. maximum time of your job) you have requested in the header of your PBS script, the batch scheduler will put your job in an appropriate waiting queue.

You can obtain information about the status of all jobs by typing one of the following commands:

In particular, the qstat command should tell you if your job is waiting to run (Q), running (R), on hold (H) or finishing (E).

If your job does not appear in the output of qstat, it might either have already been processed, or have failed because of a wrong PBS script. You could then run the tracejob command in order to have information on your job status:

Once your job has finished running (or crashed), a log file ending in .o + jobID should be written in the directory from which you submitted your job with qsub, for instance myJob.o123456.

Finally, the command qstat can be used with specific options to get more details about a finished job:

 

6 – Try it yourself!

 

You should now be able to run your first computing jobs!

  • To follow the example showing you how to compile and run a program from source code, download and extract the corresponding zip archive.
  • If you wish to use your own code, try to replicate the steps in 6.1 – Using a dummy parallel C code, or if you want to try using an existing software, refer yourself to 6.2 – Running an existing software.

 

6.0 – Common part

 

  1. First, connect to HPCaVe and create a directory to host your test codes, data and outputs. If you are using Windows, you can use a graphical client such as MobaXTerm for this purpose. On linux and MacOSX, type in a terminal:

    Note that you can either logout with the logout command, or by typing ctrl + D.

  2. Transfer necessary files from your computer to HPCaVe servers. Windows users can use Filezilla for instance to copy the archive source code and scripts, while linux and MacOSX users can type in a terminal:

  3. You can now connect back to HPCaVe servers, and check that your files have properly been copied to the hpcaveTest directory.

 

6.1 – Using a dummy parallel C code (using OpenMP on MeSU-alpha)

In this section, we are going to compile and run a dummy OpenMP parallel code on MeSU-alpha, which will approximate the value of pi in parallel according to the Monte Carlo method.

Once the source code has been transfered to HPCaVe servers, navigate to the wiki/openmp directory, and first compile the source code with gcc, enabling the fopenmp flag to enable OpenMP support:

You should now have an executable file, called pi_openmp, which we will submit to PBS through the script file called script_openmp.sh (this script will use /home as working space):

In order to launch the job on MeSU-alpha, you will have to run the command:

If the servers are not fully loaded, your job should run quite quickly, but remember that you can still check its execution status with qstat and qstat -fx jobID.

Once your job has been processed, a file called log.txt should have been written in the directory wiki/openmp, containing the result of your computation (the approximation of pi), as well as information on the time spent running.

Feel free to experiment and change the submission parameters or pieces of the code, as well as running on MeSU-beta instead of MeSU-alpha (qsub -qbeta).

Note: Other samples are provided in the archive, to have a quick preview of working with MPI and hybrid MPI/OpenMP.

Keep in mind that when using mpi, you will have to compile your code with an adapted compiler such as mpicc, and run it with mpirun or mpiexec in the PBS script:

6.2 – Running an existing software

The workflow to run an existing software (gromacs in this example) is simpler than compiling and running your own source code.

After having transfered your data, you should just have to create an appropriate PBS script file containing the commands allowing you to run the software. For instance, to run gromacs on 48 cores of MeSU-beta and on data contained in the /work directory:

And finally, to submit your script file to PBS:
At the end of the execution (checked with qstat), you should have some new data in the /work/user directory, and some information on your job in the log.txt file.