Issue while pulling centos7 image (Baobab)

HPC Support HPC issues
1

Primary information

Username: franchel
Cluster: Baobab

Description

Failing to install the ATLAS Centos7 image via apptainer on Baobab (on Yggdrassil the same procedure works smoothly).

Steps to Reproduce

  1. Connect to the interactive node cpu007
  2. Set the tmp dir to scratch area, since /tmp seems full:
    cd ~/scratch && mkdir tmp && export APPTAINER_TMPDIR=$PWD/tmp
  3. pull the image:
    apptainer pull centos7.sif docker://atlas/centos7-atlasos

Expected Result

Create the image file centos7.sif to start working on a container

Actual Result

Failing with the following messages:

(baobab)-[franchel@cpu007 scratch]$ apptainer pull centos7.sif docker://atlas/centos7-atlasos
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob 5b919389ac33 done  
Copying blob 85ec7c549d38 done  
Copying config 47f0351340 done  
Writing manifest to image destination
Storing signatures
2023/08/14 09:33:59  info unpack layer: sha256:85ec7c549d38c7af6de26ce458850556121af2bf61346dea023cd8354aa47d93
2023/08/14 09:39:36  info unpack layer: sha256:5b919389ac3342fa632b9dba4aa5951a3baadd4c1e89b6c1668f6e307e1583b1
2023/08/14 09:42:56  info verified reader: 236525446 bytes of trailing data discarded from vrdr[/home/users/f/franchel/scratch/tmp/bundle-temp-772866084/blobs/sha256/5b919389ac3342fa632b9dba4aa5951a3baadd4c1e89b6c1668f6e307e1583b1]
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: unpack entry: opt/lcg/binutils/2.37-355ed/x86_64-centos7/x86_64-pc-linux-gnu/bin/ar: link: unpriv.link: unpriv.wrap target: operation not permitted

Instead, as previously mentioned, the same procedure works on Yggdrasil.

Thanks in advance for any help!

Best,
Stefano

2

Hi @Stefano.Franchellucci

I can confirm it isn’t working when you specify to use BeeGFS home or scratch as tmpdir. The reason is probably because BeeGFS isn’t fully POSIX, at least they didn’t supported the creation of cross link hard until… now. We are still using BeeGFS 7.2.10 but we plan to upgrade to version 7.3.x or 7.4.x in the next months.

Someone has the same issue here: Cannot create image when setting SINGULARITY_TMPDIR and SINGULARITY_CACHEDIR to BeeGFS network filesystem. · Issue #1476 · apptainer/singularity · GitHub and the recommendation (our our as well) is NOT to use home or scratch as tmpdir.

Instead use /scratch or /tmp on the compute node, they are way faster for that purpose. Or even /dev/shm which is faster but will “consume” the memory you allocated for your job.

Right now it is empty:

(baobab)-[sagon@cpu007 scratch]$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       219G  9.5G  209G   5% /

Maybe someone was using it while you tried? All your temporary files on this space are deleted once you close your session.

One final word: you said it is working on Yggdrasil: you can as well simply copy the centos7.sif image to Baobab using rsync, it is probably faster anyway :grinning:

1 Like
3

Hello @Yann.Sagon ,

Thanks a lot for all the explanations. Indeed, trying again now using the compute node’s default /tmp folder did work. It did run much faster than the previous attempt on the personal ~/scratch directory.

Thanks again for the support!

Best,
Stefano