🪴 Anil's Garden

Slurm Usage Guidelines for SARDINE Servers · deep-spinwiki Wiki

02 Apr 20257 min read

Slurm Usage Guidelines for SARDINE Servers · deep-spin/wiki Wiki

Excerpt

Contribute to deep-spin/wiki development by creating an account on GitHub.

Table of Contents

  1. Introduction
  2. Launching Jobs
  3. Job Submission Methods
  4. Slurm Flags
  5. Monitoring Jobs
  6. Updating Jobs
  7. Canceling Jobs
  8. Examining Jobs

Introduction

Methods to Launch Jobs

Slurm can be used to launch new jobs in three ways:

  • srun --pty to run a shell (interactive mode)
  • srun to run a single command
  • sbatch to run a full script

Checking Job Status

Once a slurm job is launched, it will be available in the slurm queue:

IMPORTANT: After launching a job, always check its status with squeue. Sometimes you may think your job was launched and it will run shortly, but due to an unexpected error it will remain on the queue indefinitely. In such cases, you should get an error message from squeue.

Note: Slurm calculates this field only once per minute so it might not contain a meaningful value right after submitting a job.

Launching jobs

Important parameters

When launching new jobs, there are three important parameters to specify:

  • --gres=gpu:n the number of gpus required to that job (do not use --gpus=n to request gpus unless you know what you are doing, this split allocated gpus across different nodes. see here)
  • --time=[days]-hh:mm:ss a time estimate required to that job
  • --qos the name of a QOS (quality of service), which specifies the properties of the job
  • -w the name of the node to run the script. Important: At this moment, the filesystem is not shared between nodes. This means that if you have your code on poseidon, you will need to execute your script on poseidon. Thus, you should use this flag to specify that. We are working on sharing the filesystem so that you can run your code on any available node.

Quality of Service (QoS) Options

The available QOS options are:

nameprioritymax jobsmax gpusmax time
gpu-debug101801:00:00
gpu-short102404:00:00
gpu-medium52448:00:00
gpu-long222168:00:00
gpu-hero188∞

The gpu-hero QOS is only available to admins for testing purposes.

Job Submission Methods

Running a simple script with srun

The simplest way to run a script with slurm is using srun. You specify all flags through the command line and then the command you want to execute. For example, the following command allocates one gpu on poseidon for 1 hour:

srun --time=01:00:00 --gres=gpu:1 --qos=gpu-debug -w poseidon python3 your_script.py

Using tmux sessions to leave jobs running

The previous option will launch a job associated as if you would have launched it in your shell. Therefore, when you close your ssh session, your job will stop. You can use tmux to avoid this. First, launch tmux. Inside tmux, launch jobs using srun as before. In this way, your job will keep running after you log out. Here, we execute the command inside a tmux session so we can leave and the script will run.

# in artemis, launch a tmux session
$ tmux
 
# in the tmux session, prepare your job and launch a new slurm job
$ cd myproject
$ source myenv/bin/activate
$ srun --time=04:00:00 --gres=gpu:1 --qos=gpu-long -w poseidon python3 your_script.py

Debugging with interactive jobs using srun --pty

The previous alternatives are good to launch a job and forget about it until it finishes. However, you sometimes want to more closely follow your job. Interactive jobs are useful when a job requires some sort of direct intervention, for example when you want to debug. Here, we allocate a new shell where you have access to your requested resources. Inside it, you can run your programs as usual. For example: python3 your_script.py. To exit, press CTRL-D or type exit in your terminal:

srun --time=01:00:00 --gres=gpu:1 --qos=gpu-debug -w poseidon --pty bash

Running complex scripts with sbatch (advanced)

Slurm offers a way to run entire scripts with many options using sbatch. Create a script like this, called test.sbatch:

#!/bin/bash

#SBATCH --job-name=my_script    # Job name
#SBATCH --output="job.%x.%j.out # Name of stdout output file (%x expands to job name and %j expands to %jobId)
#SBATCH --time=01:00:00         # Run time (hh:mm:ss) - 1.5 hours
#SBATCH --gres=gpu:1            # Number of GPUs to be used
#SBATCH --qos=gpu-debug         # QOS to be used

python3 your_script.py

Then run it with:

sbatch -w poseidon ./test.sbatch

Check that it was submitted using

Once it ran through it should show a job.X.out file containing the job’s output.

Using Jupyter Notebooks with Vscode

Launch a tmux session within the Vscode Terminal and request for the required gpu(s) using:

srun --gres=gpu:1 --qos=gpu-long -w artemis --pty bash

Activate your desired environment and launch the notebook using:

jupyter notebook --port <port_num> --no-browser

Choose “Select Another Kernel” from the notebook you wish to run (within Vscode interface) and then go to “Existing Jupyter Server”. Specify the link generated via the above command in the prompt.

Note that the above works by itself when requesting gpu(s) from Artemis.

Slurm Flags

Expanding your command with flags can provide better control over job management and resource allocation.

FlagInfo
--job-name=nameName of the job.
--output=filenameFilename patterns may use the following place-holders such as %x for Job name and %j for Job ID (more here).
--time=timeWallclock time limit of the job. Times may be specified as [days-]hours:minutes:seconds. Example: --time=1-12:00:00 for a 36h job.
--mem=size[KMGT]Memory requirement per node. Example: --mem=4G for 4GB.
--cpus-per-task=nNumber of CPUs needed per task. Useful for multi-threaded applications. Default is one CPU per task.
--gres=gpu:1Gres specifies your resources. In this case, it specifies the requested number of GPUs.

Note that slurm automatically sets the following environment variables:

  • $SLURM_JOB_NAME (name of the job)
  • $SLURM_JOB_ID (ID of the job).

Monitoring Jobs

To get information about running or waiting jobs use

The command squeue displays a list of all running or waiting jobs of all users.

The (in our opinion) most interesting additional field is START_TIME which for pending jobs shows the date and time when Slurm plans to run this job. It is always possible that a job will start earlier but not (much) later.

Pretty output

For a prettier squeue output, you may want to add this to your ~/.bashrc:

export SQUEUE_FORMAT="%.7i %9P %35j %.8u %.2t %.12M %.12L %.5C %.7m %.4D %R"

And activate it source ~/.bashrc.

Updating Jobs

You can change the configuration of pending jobs with

scontrol update job jobid SETTING=VALUE [...]

To find out which settings are available we recommend to first run

If then for example you want to change the run-time limit of your job to let’s say three hours you would use

scontrol update job jobid TimeLimit=03:00:00

Canceling Jobs

To delete pending or running jobs you have to look up their numerical job identifiers aka job-ids. You can e.g. use squeue or squ and take the value(s) from the JOBID column. Then run

scancel -u <your_username> <job_id> [...]

and the corresponding job will be removed from the wait queue or stopped if it’s running. You can specify more than one job identifier after scancel.

If you want to cancel all your pending and running jobs without being asked for confirmation you may use

squeue -h -o %i | xargs scancel -u <your_username>.

Those options tell squeue to only output the JOBID column (-o %i) and to omit the column header (-h). Then we pipe those values as arguments to scancel.

Examining Jobs

sacct lets you examine your pending, running, and finished Slurm jobs in much more detail than the job statistics page.

$ sacct
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- --------

To view a specific job, run sacct --jobs=job_id.

References

Graph View

Backlinks

  • No backlinks found