Running jobs on Clusty

Clusty employs SLURM for resource managing and job scheduling. This service makes sure that all resources available to Clusty are shared and all jobs submitted to Clusty scheduled fairly.

Clusty is sporting a 64-core Intel mainframe, 8 72-core Intel compute nodes, 1 64-core AMD compute nodes, and 4 48-core AMD compute nodes. Each node has 1GB of memory per core. Clusty is running linux. Two UPS systems provide ~1 hour of uninterrupted power under full load and ~6 hours under throttled load in case of power outage. Disk storage is provided by a 12-unit NAS running RAID5, along with nightly backups of the /home partition to an offsite backup server. There is no provision for sensitive/restricted data. Clusty is located in a dedicated server room (nicknamed Rumble room) that is cooled to 15C by two industrial-grade air conditioning units.

To submit a job to Clusty, you need to write a SLURM shell script. The table below lists the most useful directives.

Option:
Description:
#SBATCH -J jobnameAssign the name of the job.
#SBATCH -p bigAssign the partition (queue) for the job.
#SBATCH -w nodelistRequest particular nodes. The list can be comma-separated or range, i.e. node[1-3]
#SBATCH -N numRequest num of nodes for the job.
#SBATCH -n numRequest num of processors for the job.
#SBATCH -t d-hh:mm:ssRequest walltime for the job.
#SBATCH -D dirnameSet dirname as the working directory. Relative to cwd, or absolute.
#SBATCH -o file.%j.outAssign file.%j.out as stdout for the job %j.
#SBATCH -e file.%j.outAssign file.%j.out as stderr for the job %j.
#SBATCH --mail-type=BEGIN,END,FAILNotify by mail on job start, completion or failure.
#SBATCH --mail-user=your.email@somewhere.netThe recepient of email notifications.


NOTE: the longest wall-time allowed is 2 (two) days. 48 hours. That's all you get at one time. If the process exceeds 2 days, it will be interrupted. Killed. kill -9'd. Annihilated. X-d. Nuked. Take your pick. 2 days.

A typical example for a SLURM script (run.sh) would be:

#!/bin/bash

#SBATCH -J test_job
#SBATCH -p big
#SBATCH -N 1
#SBATCH -n 32
#SBATCH -t 0-00:01:00
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=me@somewhere.net

mpirun python my_mpi_job.py

This delegates a job to one node and requests 32 processors. It also sets the maximum wall time of 1 day.

To submit this job to Clusty, use the sbatch command:

sbatch run.sh

The table below lists the most common commands for managing and monitoring your job:

Command:
Description:
sinfo Provides information on cluster state, grouped by partitions
sinfo -leN Provides a bit more information on cluster state, by machine type
squeue List all jobs in the queue
smap Curses frontend to jobs, partitions and configuration
sview Graphical frontend to jobs, partitions and configuration


The available compute nodes are:

Node:
Partition:
Cores:
Characteristics:
mainframe --- 64 Reserved for administrative use
node1 big 72 Intel E5-2695v4
node2 big 72 Intel E5-2695v4
node3 big 72 Intel E5-2695v4
node4 big 72 Intel E5-2695v4
node5 big 72 Intel E5-2695v4
node6 big 72 Intel E5-2695v4
node7 big 72 Intel E5-2695v4
node8 big 72 Intel E5-2695v4
node9 big 64 AMD Opteron 6376
node10 big 48 AMD Opteron 6344
node11 big 48 AMD Opteron 6344
node12 big 48 AMD Opteron 6344
node13 big 48 AMD Opteron 6344


Some interesting situations:

Q: Running a process on a dedicated node is far more efficient than across nodes. Can I request a whole node to myself even though I might not need all the processors available on that node?
A: Yes, pass the --exclusive switch to sbatch/srun, or include #SBATCH --exclusive in the startup script.
Q: My process is running a long time, and I would like occasional diagnostics to be printed out. It works, but it outputs my diagnostics in the end instead of as it goes along.
A: This is because of the I/O buffering. Prefix your script with stdbuf -oL -eL for line buffering, or stdbuf -o0 -e0 for character buffering.
Q: This is so awesome! What can I do to make it up to you?
A: I accept Swiss chocolate, Belgian beer and Californian full-bodied red wine donations. ;)

How to contact me? Email me!

Finally, stuff that probably doesn't interest you as a user but might interest you as a prospective maintainer.