SLURM timeout on Baobab 17.01.2019-18.01.2019

Since 17.01.2019, you may have experienced issue with job submissions on Baobab.

The symptom was a non response from slurm commands and this kind of error messages:

slurm_load_jobs error: Unable to contact slurm controller (connect failure)
slurm_load_jobs error: Socket timed out on send/recv operation

The reason is that a very high number of jobs was submitted to Baobab, ~40k.

I remind you that if you have a lot of similar jobs, the best practice is to group them in a job array.

I have modified the slurm configuration to try to mitigate this issue and it seems it’s working fine like that.

The following parameters where modified in slurm.conf:


Best regards