MPI unable to mkdir on scratch

HPC Technical
1

Hi all,

I am facing an issue that I don’t know how to solve. The essence of the code is that I use MPI to instantiate a Singularity container that then runs a piece of code that converts a .xml file into a binary file. The code looks like

mpirun --allow-run-as-root -n 1 singularity exec --bind /home/users/r/radhakrb/scratch/points_and_lines/:/usr/local/share/sdpb/ sdpb_2.5.1.sif pvm2sdp $binaryPrecision /usr/local/share/sdpb/$input /usr/local/share/sdpb/$outDir/$outFile

I guess there is a bunch of stuff in the second part of the line that might be difficult to read . But, the main parts of it is that I am mounting a specific directory in scratch inside the Singularity container so that it can then use files there to do its work.

Now unfortunately, running this piece of code gives me the following error

[cpu001.baobab:85985] PMIX ERROR: ERROR in file ../../../../../../src/mca/gds/ds12/gds_ds12_lock_pthread.c at line 168
--------------------------------------------------------------------------
A call to mkdir was unable to create the desired directory:

  Directory: /scratch
  Error:     Read-only file system

Please check to ensure you have adequate permissions to perform
the desired operation.

I am a bit surprised since I have run other similar pieces of code using MPI and Singularity without any issues (they didn’t use mkdir but they did create and write to files on scratch). It would be great if anyone has a clue as to where I am going wrong

2

Hi,

you are mounting your scratch space here /usr/local/share/sdpb/.

The error is talking about directory /scratch which isn’t the same place. The latter is a local space on every node that is thrashed when the job is finished. Mpi is probably trying to create a directory there because the variable TMP point there. You should probably mount this space as well or override the TMP variable.

3

Thanks and sorry for the late reply. Indeed, the solution was to deal with TMPDIR. Adding an unset TMPDIR in the job submission file resolves the problem. As far as I understand, it lets Singularity then use the default SINGULARITYENV_TMPDIR that was set when the .sif image was built.