Thank you for performing the maintenance and making the cluster available since today 28th August. Howoever I am facing some public internet connection problems on baobab.
I am running some gpu jobs on the cluster that I launched some hours ago at ealy afernoon, which require a public internet connection to upload metrics of my experiments and some images of simulation on wandb. I am noticing that my logs are being saved locally in my directory on the cluster.
Howoever the connection is not stable or is being cut, I believe. I see that my jobs are still running, but on the wandb platform i see that logs are stopped. When launching my jobs they connect to my wandb user logs few logs, but after few minutes I see the no logs on wandb, and the wandb jobs are being automatically stopped. But on the other side, inspecting my jobs on the cluster, they are all still running.
I also notice the following error logged in the logging file :/home/users/s/singhgu3/grey-box-interpolants multirun/2026-08-28/16-04-47/.submitit/12096158_2/12096158_2_0_log.out
--- Logging error ---
--- Logging error ---
Can you please confirm me that there is stil some connection issue on baobab?
Update 31-08: For reference, I talked with other members of our group and their jobs are also all failing. It seems to be a systematic issue since Baobab came back online.
Yes, I am also facing this error. I am trying to load a model from PyTorch using pytorch.load(...)
I am having the exact same error I/O: [Errno 121] Remote I/O error.
“”"
Evaluating outputs/swe/2b5f4757ab806d37135d863f38907365/st-inr-vf/black_box/b0abd8f3a588687e1279150830b19171/seed_4/best_chkpt.pth
Batch size for evaluation: 15, Number of batches: 7
skipped ([Errno 121] Remote I/O error)
“”"
WARNING: skipping mount of /srv/beegfs/scratch/users/s/strassep/output: stat /srv/beegfs/scratch/users/s/strassep/output: communication error on send
INFO: Terminating squashfuse_ll after timeout
INFO: Timeouts can be caused by a running background process
FATAL: container creation failed: mount hook function failure: mount /srv/beegfs/scratch/users/s/strassep/output->/srv/beegfs/scratch/users/s/strassep/output error: while mounting /srv/beegfs/scratch/users/s/strassep/output: while getting stat for /srv/beegfs/scratch/users/s/str
assep/output: stat /srv/beegfs/scratch/users/s/strassep/output: communication error on send
srun: error: gpu018: task 0: Exited with exit code 255
We have updated a system parameter, and the storage issue should now be resolved.
Please let us know if you experience any further I/O performance or stability issues.