Best practices for job scripts

Following a few guidelines can help to make your job scripts more robust and reusable. In this tip we annotate a simple script with some advice and provide pointers towards other useful documentation.

Line 1: Job scripts are assumed to be written in Bash. Making this explicit via #!/bin/bash enables text editors to treat the file appropriately (amongst other benefits).

Line 2: The minimum required of a job script - assuming you don’t specify your resource requirements entirely via qsub - is a #PBS directive.

• To reduce queuing time you should request the minimum resources necessary to run your job. See our Job sizing guidance for further information.

Line 4: Using set to configure Bash appropriately can help to avoid common issues - these flags ensure that your job will fail as soon as error occurs in your job script and that you don’t reference unset environment variables.

• Another tool that can help you verify more complex job scripts is shellcheck (conda install shellcheck).
• If you need to test a job script then you can either submit it to the debug queue (walltime=00:30:00) or run it directly as a shell script within an interactive job (qsub -I …).
• If you find yourself running qsub from within a job script then it may be a sign that you should be using array or dependent jobs instead.
• If you do find your job scripts becoming unmanageable then it may be a sign that you should be using a more appropriate workflow tool such as Snakemake or Nextflow.

Line 6: The queuing system provides you with access to a several useful environment variables, including PBS_O_WORKDIR: the directory from which you submitted the job.

• You should always try to use variables such as PBS_O_WORKDIR and HOME rather than hard-coded filesystem paths (e.g. /rds/general/user/mwoodbri/home). This makes it much easier to share and re-use job scripts. One exception is that environment variables can’t be used in directives (e.g. #PBS -e …).
• Other variables include PBS_ARRAY_INDEX (for array jobs), PBS_JOBNAME (from qsub -N …) and PBS_JOBID (e.g. 2154766.pbs).
• Note that in general it is not beneficial to “stage” your working data to and from the job’s temporary working directory (TMPDIR).
• NCPUS is available if you specify ompthreads in your resource specifier. This is only necessary in a limited number of circumstances (e.g. parallel MATLAB).
• If you need to pass any other environment variables to your job then you can use qsub -v …. This shouldn’t generally be necessary.

Line 8: As with any other software development, documenting scripts via comments (# …) can help to explain their purpose and operation to you and others.

Line 12: If you have any tasks that need to be executed before your job completes then you can use timeout to ensure that time is reserved in order to do so (5 minutes in this case). These tasks could include summarising results or tidying up ephemeral files. In general it is preferable to use job dependencies for this purpose.

Line 15: echo $SECONDS is an easy way to log how long your job took to run. It complements the memory and CPU usage metrics that are automatically logged to your job’s .o log file (see Controlling logging for jobs for further advice).

Further resources

• Getting started on the RCS web pages
• Our Beginner’s Guide to HPC course
• RCS cluster scripts on GitHub