Slurm Workload Manager - Overview
- https://slurm.schedmd.com/overview.html
- Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. Slurm requires no kernel modifications for its operation and is relatively self-contained. As a cluster workload manager, Slurm has three key functions. First, it allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. Second, it provides a framework for starting, executing, and monitoring work (normally a parallel job) on the set of allocated nodes. Finally, it arbitrates contention for resources by managing a queue of pending work. Optional plugins can be used for accounting, advanced reservation, gang scheduling (time sharing for parallel jobs), backfill scheduling, topology optimized resource selection, resource limits by user or bank account, and sophisticated multifactor job prioritization algorithms.
Slurm Workload Manager - Quick Start User Guide
Slurm Workload Manager - Wikipedia
- https://en.wikipedia.org/wiki/Slurm_Workload_Manager
- The Slurm Workload Manager (formerly known as Simple Linux Utility for Resource Management or SLURM), or Slurm, is a free and open-sourcejob scheduler for Linux and Unix-likekernels, used by many of the world's supercomputers and computer clusters.
- It provides three key functions:
- allocating exclusive and/or non-exclusive access to resources (computer nodes) to users for some duration of time so they can perform work,
- providing a framework for starting, executing, and monitoring work (typically a parallel job such as MPI) on a set of allocated nodes, and
- arbitrating contention for resources by managing a queue of pending jobs.
- Slurm is the workload manager on about 60% of the TOP500 supercomputers.[1]
- Slurm uses a best fit algorithm based on Hilbert curve scheduling or fat tree network topology in order to optimize locality of task assignments on parallel computers.[2]
Slurm Workload Manager - sacct
sbatch - Submit a batch script to Slurm
- https://slurm.schedmd.com/sbatch.html
- $ sbatch mytestsbatch.sh
- Actually the second srun will start only when the previous srun is completed, so no sleep is required.
1 # ============================================================================= 2 # mytestscript.sh 3 # ============================================================================= 4 #!/bin/sh 5 date & 6 7 # ============================================================================= 8 # mytestsbatch.sh 9 # ============================================================================= 10 #!/bin/sh 11 #SBATCH -N 2 12 #SBATCH -n 10 13 14 srun -n10 -o testscript1.log mytestscript.sh 15 sleep 10; srun -n10 -o testscript2.log mytestscript.sh 16 wait
scancel - Used to signal jobs or job steps that are under the control of Slurm.
- https://slurm.schedmd.com/scancel.html
- $ scancel 123
scontrol - view or modify Slurm configuration and state.
- https://slurm.schedmd.com/scontrol.html
- $ scontrol show job 123
squeue - view information about jobs located in the Slurm scheduling queue.
- https://slurm.schedmd.com/squeue.html
- $ squeue
- $ squeue -u username
srun - Run parallel jobs
- https://slurm.schedmd.com/srun.html
- $ cat testscript.sh
- #!/bin/sh
- python mytest.py --arg test
- $ chmod +x testscript.sh
- $ srun -N5 -n100 testscript.sh
- Run it on 5 nodes with 100 tasks
- $ srun -n5 --nodelist=host1, host2 -o testscript.log testscript.sh
- $ srun -n10 -o testscript.log --begin=now+2hour testscript.sh
- $ srun --begin=now+10 date &
Convenient SLURM Commands | FAS Research Computing
srun: error: --begin is ignored because nodes are already allocated.
- use sleep in lieu of --begin
- bash - Can you help me run tasks in parallel in Slurm? - Stack Overflow
srun: error: Unable to create job step: More processors requested than permitted
- In the submission script, you request resources with the
#SBATCHdirectives, and you cannot use more resource than than in the subsequent calls tosrun. - slurm - Questions on alternative ways to run 4 parallel jobs - Stack Overflow