# BeeGFS scratch failure

**URL:** <http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378>\
**Category:** HPC issues\
**Tags:** bamboo\
**Created:** [September 8, 2026, 8:26am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378 "2026-09-08T08:26:49Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![Josep.Garnicacaparros](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/j/f04885/32.png) [@Josep.Garnicacaparros](http://hpc-community.unige.ch/u/Josep.Garnicacaparros)\
**Post date:** [September 8, 2026, 8:26am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/1 "2026-09-08T08:26:49Z")

</div>

## Primary informations

Username: garnicac  
Cluster: bamboo

## Description

I am having problems reading and writing files in my BeeGFS scratch directory.

I am unable to reliably read files from this directory or perform Git operations on a repository located there. The filesystem returns `Remote I/O error`.

The problem affects both regular files and files inside the `.git` directory.

## Steps to Reproduce

1. Log in to the cluster and navigate to my repository:

```auto
cd /srv/beegfs/scratch/users/g/garnicac/ISC_benchmark_reproducibility

```

1. Try to copy a 651 MB CSV file from the scratch directory:

```auto
cp /srv/beegfs/scratch/users/g/garnicac/ISC_benchmark_reproducibility/ISC_benchmark/results_dev/all_results.csv ~/all_results.csv

```

1. Try reading the file with `dd`:

```auto
dd if=/srv/beegfs/scratch/users/g/garnicac/ISC_benchmark_reproducibility/ISC_benchmark/results_dev/all_results.csv \
   of=/dev/null \
   bs=1M \
   status=progress

```

1. Git operations on the same repository also fail. For example:

```auto
git commit -m "increase running time"

```

## Expected Result

I expect to be able to read and copy files from my scratch directory and perform normal Git operations on the repository.

## Actual Result

The file exists and its metadata can be read:

```auto
-rw-r--r-- 1 garnicac hpc_users 651M Sep 5 06:57 all_results.csv

```

However, copying it fails with:

```auto
cp: error reading '.../all_results.csv': Remote I/O error

```

Reading with `dd` fails after successfully reading only 1 MiB:

```auto
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied
dd: error reading '.../all_results.csv': Remote I/O error

```

Git operations are also affected. For example:

```auto
error: error reading from .git/objects/pack/pack-b6e2ddce0c1a770670404521223fb69402d1f20d.pack: Remote I/O error
error: invalid object 100644 e983947f8fdcfafb07d6806ce77fdc063416ff34 for 'ISC_benchmark/_targets.yaml'
fatal: could not open '.git/COMMIT_EDITMSG': Remote I/O error

```

There is also a broken local Git reference:

```auto
warning: ignoring broken ref refs/heads/master

```

and attempting to read `.git/refs/heads/master` gives:

```auto
cat: .git/refs/heads/master: Remote I/O error

```

Could you please check the BeeGFS filesystem/storage targets affecting my scratch directory and repair the issue? I am currently unable to reliably read or write files in this location.

---

<div class="post-metadata">

**Author:** ![Maroun.BouSleiman](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/m/8e8cbc/32.png) [@Maroun.BouSleiman](http://hpc-community.unige.ch/u/Maroun.BouSleiman)\
**Post date:** [September 8, 2026, 9:14am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/2 "2026-09-08T09:14:28Z")

</div>

also experiencing similar issues with scratch on bamboo

---

<div class="post-metadata">

**Author:** ![Maroun.BouSleiman](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/m/8e8cbc/32.png) [@Maroun.BouSleiman](http://hpc-community.unige.ch/u/Maroun.BouSleiman)\
**Post date:** [September 8, 2026, 11:57am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/3 "2026-09-08T11:57:48Z")

</div>

it seems the issue is gone. not 100% sure though.

---

<div class="post-metadata">

**Author:** ![Dan.Celestini](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/d/e99b99/32.png) [@Dan.Celestini](http://hpc-community.unige.ch/u/Dan.Celestini)\
**Post date:** [September 8, 2026, 12:01pm UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/4 "2026-09-08T12:01:54Z")

</div>

Not for me yet unfortunately.

---

<div class="post-metadata">

**Author:** ![Josep.Garnicacaparros](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/j/f04885/32.png) [@Josep.Garnicacaparros](http://hpc-community.unige.ch/u/Josep.Garnicacaparros)\
**Post date:** [September 8, 2026, 12:03pm UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/5 "2026-09-08T12:03:38Z")

</div>

not solved on my side

---

<div class="post-metadata">

**Author:** ![Maroun.BouSleiman](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/m/8e8cbc/32.png) [@Maroun.BouSleiman](http://hpc-community.unige.ch/u/Maroun.BouSleiman)\
**Post date:** [September 8, 2026, 12:25pm UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/6 "2026-09-08T12:25:38Z")

</div>

not for me either, I was wrong

---

<div class="post-metadata">

**Author:** ![Adrien.Albert](http://hpc-community.unige.ch/user_avatar/hpc-community.unige.ch/adrien.albert/32/188_2.png) [@Adrien.Albert](http://hpc-community.unige.ch/u/Adrien.Albert)\
**Post date:** [September 8, 2026, 1:25pm UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/7 "2026-09-08T13:25:12Z")

</div>

Dear HPC users,

An incident is ongoing on bamboo scratch, we are investigating:

> **[Bamboo - Storage scratch issue - HPC - Status](https://status.hpc.unige.ch/incidents/7)**
>
> We are currently investigating an issue affecting the Bamboo scratch storage system. Further updates will be provided as soon as more information becomes available.
> 
> HPC Team

---

<div class="post-metadata">

**Author:** ![Dan.Celestini](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/d/e99b99/32.png) [@Dan.Celestini](http://hpc-community.unige.ch/u/Dan.Celestini)\
**Post date:** [September 17, 2026, 7:45am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/8 "2026-09-17T07:45:52Z")

</div>

I am back to having the exact same issue since the end of the maintenance. Is this normal?

---

<div class="post-metadata">

**Author:** ![Josep.Garnicacaparros](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/j/f04885/32.png) [@Josep.Garnicacaparros](http://hpc-community.unige.ch/u/Josep.Garnicacaparros)\
**Post date:** [September 17, 2026, 7:51am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/9 "2026-09-17T07:51:15Z")

</div>

same problem on my side

---

<div class="post-metadata">

**Author:** ![Maroun.BouSleiman](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/m/8e8cbc/32.png) [@Maroun.BouSleiman](http://hpc-community.unige.ch/u/Maroun.BouSleiman)\
**Post date:** [September 17, 2026, 7:57am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/10 "2026-09-17T07:57:48Z")

</div>

Also for me. I had no issues immediately after maintenance, but then experienced issues. Do you know when it started? I had some overnight stuff that failed and it would help to know if it’s me or beegfs!

---

<div class="post-metadata">

**Author:** ![Adrien.Albert](http://hpc-community.unige.ch/user_avatar/hpc-community.unige.ch/adrien.albert/32/188_2.png) [@Adrien.Albert](http://hpc-community.unige.ch/u/Adrien.Albert)\
**Post date:** [September 17, 2026, 10:22am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/11 "2026-09-17T10:22:55Z")

</div>

Dear Users,

I’m invastigating right now. I keep you informed.

---

<div class="post-metadata">

**Author:** ![Adrien.Albert](http://hpc-community.unige.ch/user_avatar/hpc-community.unige.ch/adrien.albert/32/188_2.png) [@Adrien.Albert](http://hpc-community.unige.ch/u/Adrien.Albert)\
**Post date:** [September 18, 2026, 7:29am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/12 "2026-09-18T07:29:28Z")

</div>

> **[Bamboo - Storage scratch issue - HPC - Status](https://status.hpc.unige.ch/incidents/8)**
>
> The issue has been resolved and the service has been fully restored.
> 
> Keep watching

---

<div class="post-metadata">

**Author:** ![Maroun.BouSleiman](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/m/8e8cbc/32.png) [@Maroun.BouSleiman](http://hpc-community.unige.ch/u/Maroun.BouSleiman)\
**Post date:** [September 18, 2026, 8:07am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/13 "2026-09-18T08:07:12Z")

</div>

Thank you. It seems to be working normally for me.

---

<div class="post-metadata">

**Author:** ![Adrien.Albert](http://hpc-community.unige.ch/user_avatar/hpc-community.unige.ch/adrien.albert/32/188_2.png) [@Adrien.Albert](http://hpc-community.unige.ch/u/Adrien.Albert)\
**Post date:** [September 18, 2026, 8:11am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/14 "2026-09-18T08:11:13Z")

</div>

It seems to be a hardware issue. We are still monitoring the situation, but we cannot guarantee that the issue will not occur again.

---

<div class="post-metadata">

**Author:** ![Maroun.BouSleiman](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/m/8e8cbc/32.png) [@Maroun.BouSleiman](http://hpc-community.unige.ch/u/Maroun.BouSleiman)\
**Post date:** [September 18, 2026, 8:12am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/15 "2026-09-18T08:12:27Z")

</div>

Thank you. Hope it is not too serious. Is there anything we should do or avoid doing as users? Are all scratch locations (shared vs user) equally affected?

---

<div class="post-metadata">

**Author:** ![Adrien.Albert](http://hpc-community.unige.ch/user_avatar/hpc-community.unige.ch/adrien.albert/32/188_2.png) [@Adrien.Albert](http://hpc-community.unige.ch/u/Adrien.Albert)\
**Post date:** [September 18, 2026, 8:19am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/16 "2026-09-18T08:19:29Z")

</div>

Thank you for your understanding.

At this stage, there is nothing specific users need to do. We suspect a firmware-related issue that can occasionally cause part of the scratch storage to become unavailable until the affected server is rebooted.

The issue affects the entire scratch filesystem, including both shared and user scratch areas. We are working with our hardware vendor to identify a permanent solution.

---

<div class="post-metadata">

**Author:** ![Gael.Rossignol](http://hpc-community.unige.ch/user_avatar/hpc-community.unige.ch/gael.rossignol/32/206_2.png) [@Gael.Rossignol](http://hpc-community.unige.ch/u/Gael.Rossignol)\
**Post date:** [September 21, 2026, 11:52am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/17 "2026-09-21T11:52:15Z")

</div>

Split this topic on September 21, 2026, 11:52am UTC

2 posts were split to a new topic: [Mountpoint issue](http://hpc-community.unige.ch/t/mountpoint-issue/4386)

---

<div class="post-metadata">

**Author:** ![Timea.Szalai](http://hpc-community.unige.ch/letter_avatar_proxy/v4/letter/t/ba8739/32.png) [@Timea.Szalai](http://hpc-community.unige.ch/u/Timea.Szalai)\
**Post date:** [September 24, 2026, 1:24pm UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/18 "2026-09-24T13:24:45Z")

</div>

Hi I think I also encountered a similar error today:  
I had one job which produced incomplete files and another one not producing the expected outcome, I am working from home and this occurred via the Bamboo Open On Demand shell access/ scratch. When trying to understand the problem, the “Remote I/O error” also appeared. Now impossible to launch any command.

I am very sorry if it is me who did a woopsy, but as it seems to be an ongoing problem, I thought it would be better to notify you.

---

<div class="post-metadata">

**Author:** ![Gael.Rossignol](http://hpc-community.unige.ch/user_avatar/hpc-community.unige.ch/gael.rossignol/32/206_2.png) [@Gael.Rossignol](http://hpc-community.unige.ch/u/Gael.Rossignol)\
**Post date:** [September 25, 2026, 8:57am UTC](http://hpc-community.unige.ch/t/beegfs-scratch-failure/4378/19 "2026-09-25T08:57:43Z")

</div>

Dear Timea,

You are right, yesterday we had a new occurrence of the I/O issue. The server was rebooted quickly, but you encountered some problems.

Sorry for the inconvenience,
