Submitting SLURM Jobs

Batch Job

A batch job can be launched using sbatch command and with a script. The most common options for a batch job are:

-N <#>/ --nodes=<#> -- request nodes

-n <#> / --tasks=<#> -- request tasks (e.g. cores)

-c <#> / --cpus-per-task=<#> -- request # cpus per task

--ntasks-per-node=<#> -- request # tasks/cores per node

--constraint=femto # Should be femto, cyclops or gust depending on cluster

A few example scripts are given below.

1. 64 core parallel job:

#!/bin/tcsh
#SBATCH --job-name=parallel
#SBATCH --nodes=2 --ntasks-per-node=32
#SBATCH --constraint=femto
#SBATCH -t 30:00

srun ./a.out

2. 2 node hybrid parallel job

#!/bin/tcsh
#SBATCH --job-name=hybrid
#SBATCH --nodes=2 --ntasks=2 --cpus-per-task=32
#SBATCH --constraint=femto
#SBATCH -t 30:00

ckload 0.05 # report if any node usage is greater than 0.05
srun ./a.out

3. 1 node packed with multiple serial jobs

#!/bin/tcsh

#SBATCH --job-name=array1
#SBATCH --output=job_%A_%a.out
#SBATCH --error=job_%A_%a.err
#SBATCH --time=01:00:00
#SBATCH -N 1
#SBATCH --ntasks-per-node=32

srun -n 1 --exclusive --output=job_%A_%a_1.out ./a1.out &
srun -n 1 --exclusive --output=job_%A_%a_2.out ./a2.out &
srun -n 1 --exclusive --output=job_%A_%a_3.out ./a3.out &
srun -n 1 --exclusive --output=job_%A_%a_4.out ./a4.out &
wait

4. Array job with unique inputs - Please note that argument to --input is used instead of standard input
g
#!/bin/tcsh
#SBATCH --job-name=arrayj
#SBATCH --output=array_%A_%a.out
#SBATCH --error=array_%A_%a.err
#SBATCH --time=1-1:00
#SBATCH --array=1-4
#SBATCH -N 1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32

srun -N1 -c32 --input=input%a --exclusive ./ppl.py

Interactive Job

Currently, the femto and cyclops clusters are running an older version of slurm compared to gust. These two versions differ on how to submit interactive jobs:

- Femto/Cyclops

srun can be used for a blocking interactive job with a single job-step and stdout and stderr are displayed on the screen. srun is also preferable for jobs that require X11 forwarding. To launch an interactive job via slurm run the following type of command:

>> srun -N 1 -n 32 --pty tcsh # using one node and 32 tasks

>> srun -N 1 -n 32 --x11 --pty tcsh #for X11 forwarding

- Gust

For Gust, the command should be:

>> salloc -N 1 -n 32

>> salloc -N 1 -n 32 --x11

Input/Output Formatting

Given below are some useful specifiers that can be used to distinguish outputs

String	Formatting
//	Cancel replacement
%A	Job id
%a	Array index
%x	Name