BeeGFS scratch failure

HPC Support HPC issues
1

Primary informations

Username: garnicac
Cluster: bamboo

Description

I am having problems reading and writing files in my BeeGFS scratch directory.

I am unable to reliably read files from this directory or perform Git operations on a repository located there. The filesystem returns Remote I/O error.

The problem affects both regular files and files inside the .git directory.

Steps to Reproduce

  1. Log in to the cluster and navigate to my repository:
cd /srv/beegfs/scratch/users/g/garnicac/ISC_benchmark_reproducibility
  1. Try to copy a 651 MB CSV file from the scratch directory:
cp /srv/beegfs/scratch/users/g/garnicac/ISC_benchmark_reproducibility/ISC_benchmark/results_dev/all_results.csv ~/all_results.csv
  1. Try reading the file with dd:
dd if=/srv/beegfs/scratch/users/g/garnicac/ISC_benchmark_reproducibility/ISC_benchmark/results_dev/all_results.csv \
   of=/dev/null \
   bs=1M \
   status=progress
  1. Git operations on the same repository also fail. For example:
git commit -m "increase running time"

Expected Result

I expect to be able to read and copy files from my scratch directory and perform normal Git operations on the repository.

Actual Result

The file exists and its metadata can be read:

-rw-r--r-- 1 garnicac hpc_users 651M Sep 5 06:57 all_results.csv

However, copying it fails with:

cp: error reading '.../all_results.csv': Remote I/O error

Reading with dd fails after successfully reading only 1 MiB:

1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied
dd: error reading '.../all_results.csv': Remote I/O error

Git operations are also affected. For example:

error: error reading from .git/objects/pack/pack-b6e2ddce0c1a770670404521223fb69402d1f20d.pack: Remote I/O error
error: invalid object 100644 e983947f8fdcfafb07d6806ce77fdc063416ff34 for 'ISC_benchmark/_targets.yaml'
fatal: could not open '.git/COMMIT_EDITMSG': Remote I/O error

There is also a broken local Git reference:

warning: ignoring broken ref refs/heads/master

and attempting to read .git/refs/heads/master gives:

cat: .git/refs/heads/master: Remote I/O error

Could you please check the BeeGFS filesystem/storage targets affecting my scratch directory and repair the issue? I am currently unable to reliably read or write files in this location.

2

also experiencing similar issues with scratch on bamboo

3

it seems the issue is gone. not 100% sure though.

4

Not for me yet unfortunately.

5

not solved on my side

6

not for me either, I was wrong

7

Dear HPC users,

An incident is ongoing on bamboo scratch, we are investigating:

8

I am back to having the exact same issue since the end of the maintenance. Is this normal?

9

same problem on my side

10

Also for me. I had no issues immediately after maintenance, but then experienced issues. Do you know when it started? I had some overnight stuff that failed and it would help to know if it’s me or beegfs!

11

Dear Users,

I’m invastigating right now. I keep you informed.

12
13

Thank you. It seems to be working normally for me.

14

It seems to be a hardware issue. We are still monitoring the situation, but we cannot guarantee that the issue will not occur again.

15

Thank you. Hope it is not too serious. Is there anything we should do or avoid doing as users? Are all scratch locations (shared vs user) equally affected?

16

Thank you for your understanding.

At this stage, there is nothing specific users need to do. We suspect a firmware-related issue that can occasionally cause part of the scratch storage to become unavailable until the affected server is rebooted.

The issue affects the entire scratch filesystem, including both shared and user scratch areas. We are working with our hardware vendor to identify a permanent solution.

18

Hi I think I also encountered a similar error today:
I had one job which produced incomplete files and another one not producing the expected outcome, I am working from home and this occurred via the Bamboo Open On Demand shell access/ scratch. When trying to understand the problem, the “Remote I/O error” also appeared. Now impossible to launch any command.

I am very sorry if it is me who did a woopsy, but as it seems to be an ongoing problem, I thought it would be better to notify you.

19

Dear Timea,

You are right, yesterday we had a new occurrence of the I/O issue. The server was rebooted quickly, but you encountered some problems.

Sorry for the inconvenience,