SLURM chained jobs

HPC Technical
1

Hi all,

I was curious to learn how to do chained SLURM jobs.

The bash script chain is the following. Each bash script should be executed once the prior one is complete without errors. If one step fails, the whole chain should stop.

  1. sbatch run_cost_optimal_scenario.sh
  2. sbatch --array=1-12 run_pareto_optimal_scenarios.sh
  3. sbatch --array=1-100 run_MGA_scenarios.sh
  4. sbatch run_summary.sh
  5. sbatch run_cleanup.sh
  6. sbatch run_costoptimal_LMP.sh
  7. sbatch --array=1-12 run_pareto_LMP.sh
  8. sbatch --array=1-100 run_MGA_LMP.sh

Each bash script takes in an array and passes sets of parameters to specific python scripts.
e.g. sbatch --array=1-100 run_MGA_scenarios.sh reads in array numbers 1-100 and configures parameters for scenarios, which are then forwarded to a python script → model.py.

I wasn’t able to correctly set this up yet.

Do you have any advice on this?

Thanks!
Jan

2

Hello,

are you looking for something like --dependency:afterok? Please see our doc for some information about that.

Best

3

ah yes that’s it.

thanks!

4

I was going to do something like this:
(source: Building pipelines using slurm dependencies)

#! /bin/bash

#first job - no dependencies
jid1=$(sbatch --mem=12g --cpus-per-task=4 job1.sh)

#multiple jobs can depend on a single job
jid2=$(sbatch --dependency=afterany:$jid1 --mem=20g job2.sh)
jid3=$(sbatch --dependency=afterany:$jid1 --mem=20g job3.sh)

#a single job can depend on multiple jobs
jid4=$(sbatch --dependency=afterany:$jid2:$jid3 job4.sh)

#swarm can use dependencies
jid5=$(swarm --dependency=afterany:$jid4 -t 4 -g 4 -f job5.sh)

#a single job can depend on an array job
#it will start executing when all arrayjobs have finished
jid6=$(sbatch --dependency=afterany:$jid5 job6.sh)

#a single job can depend on all jobs by the same user with the same name
jid7=$(sbatch --dependency=afterany:$jid6 --job-name=dtest job7.sh)
jid8=$(sbatch --dependency=afterany:$jid6 --job-name=dtest job8.sh)
sbatch --dependency=singleton --job-name=dtest job9.sh

#show dependencies in squeue output:
squeue -u $USER -o “%.8A %.4C %.10m %.20E”