Several nodes in drain

[Debajyoti.Sengupta]

Hi,
It appears that several nodes are currently in drain/down state.

Regards,
Debajyoti

[Adrien.Albert]

Dear @Debajyoti.Sengupta

Here the status of this incident:

[2023] Current issues on HPC Cluster

Baobab infiniband issue Dear users, Due to an incident that occurred last night on the infiniband fabric on the Baobab cluster. Compute nodes have been drained. We are actively working to resolve the problem. The nodes will gradually return to production. We apologize for the inconvenience. Thank you for your understanding. Best regards, Status : Resolved start: 2023-09-12T01:26:00Z (UTC) end: 2023-09-12T09:00:00Z (UTC) Status : Resolved Start: 20…

Computes are back into production.

Best Regards

[Malte.Algren]

They continue to be drain/down

[Adrien.Albert]

Dear @Debajyoti.Sengupta

We have identified the root cause of the issue. An infiniband switch is experiencing hardware malfunction, resulting in interference with the infiniband traffic across the entire cluster. Currently, the nodes connected to this switch have been temporarily disabled and powered off. We are awaiting the replacement hardware to restore these nodes.

We apologize for any inconvenience this may have caused.

Best Regards,

[Malte.Algren]

It might be a related issue, but it seems like Scratch is not working at the moment.

[Adrien.Albert]

Hi @Malte.Algren

it may be related, we’re working on it. We will keep you inform.

Best Regards