Unable to access CVMFS from shared-gpu

HPC Support HPC issues
1

Primary informations

Username: nussbaup
Cluster: Yggdrasl

Description

Today (Wednesday) from around 16:30 on, I was unable to access the CVMFS filesystem from nodes gpu002 and gpu003 (shared-gpu partiiton). Just one hour earlier this worked fine.

In my sbatch script I call at some point:

source /cvmfs/sft.cern.ch/lcg/views/LCG_104cuda/x86_64-centos8-gcc11-opt/setup.sh

which yields

zsh:source:1: no such file or directory: /cvmfs/sft.cern.ch/lcg/views/LCG_104cuda/x86_64-centos8-gcc11-opt/setup.sh

I then tried to simply ls the path to the setup.sh to see what would happen. This did not work on gpu002 and gpu003 (see the minimum example to reproduce below).

I tried to “ls” the setup.sh from login001, cpu001 (via debug-cpu) and cpu008 (via shared-cpu). From all three of these it worked perfectly.

Steps to Reproduce (last tested at 18:00)

salloc --partition shared-gpu --gpus 1 --time 00:02:00 --nodelist gpu002
ls /cvmfs/sft.cern.ch/lcg/views/LCG_104cuda/x86_64-centos8-gcc11-opt/setup.sh

Expected Result

I would expect the file path to be displayed:

/cvmfs/sft.cern.ch/lcg/views/LCG_104cuda/x86_64-centos8-gcc11-opt/setup.sh

Actual Result

Nothing (or cancelled due to timelimit)

2

Hi @parzival.nussbaum

The root cause of CVMFS issue is really foggy. Everything seemed working fine. The proxy server have been refreshed, and everything is working again now.

gpu002.yggdrasil:
Probing /cvmfs/euclid.in2p3.fr... OK
Probing /cvmfs/euclid-dev.in2p3.fr... OK
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/atlas-nightlies.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK

If you encounter any further issues or have questions, feel free to reach out.