Intermittent Issue with storage on Bamboo

Dear Users,

We are currently experiencing stability issues affecting both the /home and /scratch filesystems on the Bamboo cluster.

Technical Context:
While /home and /scratch are stored on different physical disks, they are hosted on the same server infrastructure. The load is distributed across 4 servers, each equipped with 4 RAID controllers and disk enclosures. The current instability appears to be linked to a potential hardware fault in one of these RAID controllers. If a controller fails, it impacts both /home and /scratch simultaneously.

Actions Taken:
We are aware that this is disruptive to your workflows. To rule out firmware-related issues, we have just updated the BIOS on all 4 servers.

We have also contacted our vendor to address the issue. We are expecting a replacement RAID controller to be delivered shortly.

Next Steps & Limitations:
To replace the faulty hardware, we will need to stop the storage services. Please note that even after the replacement, there is no guarantee that the issue will be fully resolved, as the root cause may be more complex than a single hardware component failure.

We will keep you updated as soon as we have a concrete timeline for the maintenance window or further news.

Thank you for your patience and understanding.

Best regards,

The HPC Support Team

1 Like