I tried a lot of different version of my batch, but it keeps crashing all the time. My goal is just to run fMRI preprocessing in Matlab for 74 subjects in parallel. I initially tried with a parfor loop in Matlab, then abandoned it and replaced it by a loop inside the sbatch using the --array command but whatever I do the batch crashes after around 1h without any error message but without finishing the job it was supposed to do. I don’t understand why it keeps failing, especially since I don’t get any error message in Matlab or in the batch suggesting that the problem lies in the way Slurm interacts with my script (RAM issue?)
The batch should have launched the 73 corresponding subjects in parallel and I should have my preprocessing done. Instead, it starts doing it but crashes randomly about 1h or so after the start after performing 4-6 subjects (depending on the number of available nodes) and then crashes with no further explanation right in the middle…
Actual Result
The log files indicate that the script crashes randomly but before finishing.
I managed to make it finish one subject once when asking for just 2 nodes to work in parallel as if by reducing the number of jobs it would be more efficient. I’m currently considering to remove the parallelizing (which is the main interest of the cluster) or to switch back to a local analysis as this has already cost me weeks without understanding what is going wrong so any help would be more than welcome!
Hi
Indeed the message I receive in my email box is the following:
“Slurm Array Summary Job_id=15203285_* (15203285) Name=preprocessing Ended, Mixed, MaxSignal [53]”
so I also have the code 53. However, the path is correct in my case (there is a writtable scripts/logs folder where the output log files are written without any issue). Could the problem come from the fact that in my batch the output and error log files have the same name?
In the thread you shared, someone also says “the same thing happens if you exceed your quota” which would be consistent with the fact that my batch often crashes around 1h after starting but I don’t know why I would have such a limited quota though?
I will try again by changing the error logfile name, but I doubt it comes from it as I think I already had a batch like that in another institute and it was not a problem
Not sure if it will crash or not, but I already see that this led to the creation of the corresponding error files while they were not created before. Thanks for the tip. Let’s see if that solves the issue
So it seems that there is one subject which crashes leading to the whole script to crash (not sure why Slurm doesn’t compute the others when one crashes) but now I have the error log file to see it as well
Multiple processes writing to the same file can cause data corruption and inconsistencies, so it’s not recommended. But I think SLURM just splits stderr and stdout, so it should not be an issue.
I just ran another batch (15211163_) which also crashed despite fixing the two previously mentionned issues (one subject had an issue with a path definition + the error file output now has a different name). Again everything crashed ~30min after starting. All the error logs are empty (can be found in /home/users/c/clairis/scripts/logs) and the batch script shows that all subjects were interrupted in the beginning of the analysis with no subject actually completed.
No, all jobs (i.e. all subjects) should be completely independent.
I didn’t re-run a test recently, but the only subject that could be run until the end was due to the fact that I reduced the number of subjects processed in parallel to 2. Could there be an issue related to the fact that I give the full array of 73 individuals? Should I specify a smaller number via --array=1-73%5 for example to reduce the number of subjects being processed in parallel? I don’t know why that would be an issue though as the cluster should be precisely optimized to deal with that kind of situations right?
the email message is the following: Slurm Array Summary Job_id=15211163_* (15211163) Name=wololo_preprocessing Ended, Mixed, MaxSignal [53]
so again code 53 strangely
I’m going to try to reduce the number of jobs performed in parallel with the following command: #SBATCH --array=1-73%5
and I removed any constraint on the RAM so that Slurm allocates it adaptively. Let’s see if these changes improve things, but would be nicer to be maximize the number of participants performed in parallel though.
After discussing with Adrien, I moved all my files from the /home to the /scratch folder and redid all the analysis after changing all the paths and everything seems to have worked this time. It seems that the issue was coming from the fact that the analysis was saturating the allowed space in the home folder (1To) and my script was crashing whenever that limit was reached (hence the apparent random timing of when it was crashing).
I would like to add a small clarification: from my point of view, I suspect that home storage is almost full rather than your quota limit, (both are possible):
(baobab)-[root@admin1 ~]$ df /home -h
Filesystem Size Used Avail Use% Mounted on
beegfs_home 138T 137T 1.9T 94% /home
We’ve sent an email to the main users (>500GB) inviting them to empty their homes as far as possible.