Crash with logging error since the update

HPC Support HPC issues
1

Since the update my SLURM jobs on shared-gpu fail intermittently with OSError: [Errno 121] Remote I/O error (EREMOTEIO) on writes and flushes.

  • Python processes abort with:

    OSError: [Errno 121] Remote I/O error
  File ".../logging/__init__.py", line 1144, in flush
    self.stream.flush()

    (also raised from pandas .to_csv, numpy.save, joblib.dump — i.e. any write).

  • Shell in the job script: slurm_script: line NN: echo: write error: Remote I/O error

  • SLURM records the array tasks as FAILED, ExitCode 120:0 or 1:0.

  • When it hits the process’s stdout stream, Python’s logging error handler re-enters on every failed flush, producing thousands of --- Logging error --- blocks and occasionally RecursionError: maximum recursion depth exceeded.

Can you help, please ?

Best,

Thibaut Chataing

2

It seems there is no a global problem with the disk :

(baobab)-[chataint@login1 slurm]$ ls
ls: reading directory ‘.’: Communication error on send
(baobab)-[chataint@login1 slurm]$ echo $PWD
/srv/beegfs/scratch/shares/schaerm/schaer2/video_sam2_pose/humanlisbet-code/slurm

3

Dear Thibaut,

Since the Baobab migration, we have been encountering storage issues following an update to the latest major release. You can track the status here:

We are actively working with vendor support to find a solution.

In the meantime, please feel free to use other clusters (Bamboo or Yggdrasil) until the issue is resolved.

We apologize for the inconvenience.

Best regards

4

Dear Users,

We have updated a system parameter, and the storage issue should now be resolved.
Please let us know if you experience any further I/O performance or stability issues.

Best regards,