Quotas and backups

Back-up:
While your /home directory is regularly backed-up, please keep in mind that the scratch directories don’t have any backup system.
It is therefore important to check that you DO NOT store any important data on /scratchalpha and /scratchbeta, without having a backup of them elsewhere.

Quotas:
With the default HPCaVe account, your storage space is limited according to the following quotas:

  • /home: 30Go

Scratch directories

Scratch spaces are temporary spaces for your jobs files.

Please note that /scratchalpha is only available from MeSU-alpha, while /scratchbeta can only be accessed from MeSU-beta.

In other words, if you launch a job on MeSU-alpha, you won’t be able to write or read data to/from /scratchbeta, which will simply not be visible for your job (and vice-versa).

  • /scratchalpha for Mesu-alpha is a direct attachement SAS file system physically on the UV-2000 so when your job run on that scratchalpha volume it does not go through the network at any time. Note that the performance on /scratchalpha is better than on NFS file system (such as your /home directory).
  • /scratchbeta is a Lustre filesystem, distributed through Infiniband that also has better performances than NFS filesystems. It is available on the login nodes, on MeSU-beta and on MeSU-gamma.

Using scratch directories

The scratch directories are created on your first login, and are located at /scratchalpha/$USER and /scratchbeta/$USER

If you do not tell PBS to use those spaces explicitly, your job will be executed from your current working directory (most often your home directory).

Steps to use scratch directories :

  • Compile your code if needed (you can do it in your /home directory, but also in your scratch spaces).
  • Move any input data files to the appropriate scratch directory. If you plan on using MeSU-alpha for instance, move your input files to /scratchalpha/$USER
  • If you wish to run your job directly from /scratchalpha:
    • Copy your executable and your PBS script in a dedicated directory in your scratch space.
    • Change your working directory to /scratchalpha/$USER with cd
    • Launch your script with qsub from /scratchalpha
  • If you wish to run your job from your /home directory:
    • edit your PBS script to use the appropriate environment variables and read/write from/to the correct directories
    • Launch your script from your home directory

For instance, here is a script you could launch from your /home directory, which will write in your /scratchalpha directory:

#!/bin/bash
#PBS -q alpha
#PBS -l select=1:ncpus=16
#PBS -l walltime=60:00:00
#PBS -N myTestJob

#load appropriate modules
module purge
module load intel/intel-compilers-18.0/18.0
module load modulefiles/mpt/2.16

#move to PBS_O_WORKDIR
cd $PBS_O_WORKDIR

# Define scratch space scratch alpha for UV scratchbeta for ICE
SCRATCH=/scratchalpha/$USER/myprogram_scratch_space
PROJECT=’my project name’
mkdir $SCRATCH
mkdir $SCRATCH/$PROJECT

# copy some input files to  $SCRATCH directory
cp some_input_files  $SCRATCH/$PROJECT

#execute your program
cd $SCRATCH/$PROJECT || exit 1
myprogram 1> myprogram.out 2> myprogram.err

# copy some output files to submittion directory and delete temporary work files
cp -p some_output_files $PBS_O_WORKDIR || exit 1

#clean the temporary directory
rm -rf ${SCRATCH}/${PROJECT}

And here is a sample file for MeSU-beta:

#!/bin/bash
#PBS -q beta
#PBS -l select=2:ncpus=24:mpiprocs=24
#PBS -l walltime=60:00:00
#PBS -N mytestjob
#PBS -j oe

#load appropriate modules
module purge
module load intel-compilers-18.0/18.0
module load mpt/2.18


#move to PBS_O_WORKDIR
cd $PBS_O_WORKDIR
 
# Define scratch space scratchbeta on ICE XA
SCRATCH=/scratchbeta/$USER/myprogram_scratch_space
PROJECT=’my project name’
mkdir $SCRATCH
mkdir $SCRATCH/$PROJECT

# copy some input files to  $SCRATCH directory
cp some_input_files  $SCRATCH/$PROJECT

cd $SCRATCH/$PROJECT || exit 1
mpirun -n 48 myprogram 1> myprogram.out 2> myprogram.err

# copy some output files to submittion directory and delete temporary work files
cp -p some_output_files $PBS_O_WORKDIR || exit 1
 
#clean the temporary directory
rm -rf ${SCRATCH}/${PROJECT}

Just keep in mind the quotas on homes and the absence of backups on scratch spaces. in your home must be only the results and the codes you need and your personal software (conda environnement, your own compilation of a program with the options you specify …)