Symlinks broken in /cvmfs on Baobab

HPC Support HPC issues
1

Primary informations

Username: coppinp
Cluster: Baobab

Description

There seems to be an issue with how /cvmfs is mounted on the computing nodes. All symlinks stopped broken.

Steps to Reproduce

(baobab)-[coppinp@login2 ~]$ srun -p public-interactive-cpu --time=02:00:00 --ntasks-per-node 1 --mem=2G --pty bash
srun: job 4814151 queued and waiting for resources
srun: job 4814151 has been allocated resources
(baobab)-[coppinp@cpu007 ~]$ cd /cvmfs/
(baobab)-[coppinp@cpu007 cvmfs]$ ls
cvmfs-config.cern.ch dampe.cern.ch
(baobab)-[coppinp@cpu007 cvmfs]$ cd dampe.cern.ch/
bash: cd: dampe.cern.ch/: Too many levels of symbolic links

On a 2nd attempt (not changing anything) I can suddenly ‘cd’ into this directory, but it then it shows all symlinks in the directory /cvmfs/dampe.cern.ch/ as broken.

Expected Result

At least before this week’s upgrade to Baobab everything was working correctly. Is it possible that this cvmfs directory somehow got incorrectly mounted?

2

Update: several people in our group keep getting these errors. A small fraction of jobs launched through slurm return the error Too many levels of symbolic links when sourcing a script inside the directory /cvmfs/dampe.cern.ch/

Relaunching the jobs often suffices to make it run successfully. However, the error persists and is affecting the work / jobs launched by several users.

3

Dear Paul,

During the last maintenance of Baobab we have changed a parameter on slurm : “JobContainerType=job_container/tmpfs” to use slurm default plugin instead of the third party one.

But this change introduce an issue regarding autofs mount of filesystem used by cvmfs.

On Tuesday, this last week we have done a rollback on this change and now all is working fine.

Best regards,

1 Like