Intermittent I/O Errors

Primary informations

Username: froelicm
Cluster: Bamboo

Description

Encountering I/O Errors such as read and write operations on text files in scratch. Errors seem to be intermittent, as some operations are successful. Main associated errors seem to be “Remote I/O error” and “targetID: 1101; Msg: Internal error” in the logs.

Steps to Reproduce

f=/home/users/f/froelicm/scratch/.fs_repro_test
echo "test" > "$f" && cat "$f" > /dev/null

Expected Result

Write to f and open it.

Actual Result

Exactly a 50% success rate, alternating between “Remote I/O error” and success.

bash: /home/users/f/froelicm/scratch/.fs_repro_test: Remote I/O error

Hello,

The server has been rebooted and the service is now fully operational. A configuration parameter has been adjusted to prevent this issue from happening again.

We apologize for the inconvenience.

Hi Gael,

Seems to be happening again! Same test yields the same error with 100% fail rate.

I think the issue continues,
I’m experiencing a similar problem in bamboo by reading MiniSEED files from the scratch filesystem. The files were accessible earlier, but reads are now failing intermittently.

when trying to copy data to my home:

cp: error reading ‘*.mseed’: Remote I/O error

ObsPy also promts invalid/skipped MiniSEED records when trying to read the same files.