we are currently having a couple of compatibility issues with containers and are trying to debug the source of the problem.
Would it be possible to install singularity 3.7.2-1 such that we can replicate our setup with that of another HPC site where we are succesfully able to run inside the particular singularity container giving us problems.
We have also noticed that by default we cannot run with singularity containers in write mode - is this something which can be enabled from the admin side? (Most documentation has singularity run with sudo).
thanks for the reply and looking into this!
I think we should be fine with the newer version, I can imagine the open stack machines at CERN I was comparing to will be updating to 3.7.3 soon, and having a slightly newer version will be a little easier than the current version for compatability checks! (In fact I just checked a node in pre-prod and it is on 3.7.3-1.el7).
For sudo, I haven’t come across an HPC with this enabled, and this is where I would have to refer to your expertise, but looking at the singularity documentation at least building containers will always require root privaleges (which for sure I can imagine you don’t want to give away!) [ref] though for the most part we are bootstrapping docker containers built with a CI git runner.
Another issue we had (to provide some context) is in pulling a docker container from the CERN registeries on a worker node gave the following error ERROR: build: failed to make environment dirs: mkdir /tmp/sbuild-235840019/fs/.singularity.d: permission denied FATAL: While making image from oci registry: while building SIF from layers: packer failed to pack: while inserting base environment: build: failed to make environment dirs: mkdir /tmp/sbuild-235840019/fs/.singularity.d: permission denie @Manuel.Guth had a workaround with setting chmod on the /tmp/sbuild<id> directory, though this problem was observed on another HPC as well so isn’t a fault of the configuration (and we have crosschecked the singularity.conf files across both HPC and they are pretty much identical).
I don’t know if you have any advice for the workflow, or know of another way to enable the containers themselves to run in writeable mode, but one of the issues we have had has been with eggs and singularity. We currently specify an install directory that is on the local file system and not in the container, but then it isn’t so portable. Another thing I considered was changing the format from sif to ext3, which I believe is how I had it working in the past. Our aim is to develop code locally with python setup.py develop and run it within the container.
For the toolchain I don’t think we have much of a preference, if you would have a suggestion - for us we just want to make sure we have the newest CUDA version possible to run on the A100s!
Thanks in advance for the replies and again for all your work!
To let you know and keep you in the loop, this version fixed our issues with pulling docker containers (we still have some path warnings as a rootless user but I don’t think these can be avoided without sudo), so the first point is definitely solved.