[yggdrasil] Hardware problem around 4AM 2023-01-27

Dear @support,

Was there some technical problem on Ygrassil around 4 AM tonight (2023-01-27)? I observed multiple jobs extremely slowed down (leading to TIMEOUT) around that time.

Kind regards,
Maciej Falkiewicz

Dear Maciej,

We didn’t have any trouble during this time range on Yggdrasil. Monitoring is green.

Maybe you can give me job number to investigate?

Best regards,

I already forgot about this thread :slight_smile: If you didn’t see anything exceptional on your side, this must have been the standard problem of access to storage, which crushes our jobs regularly.

Please find the list of jobs below:
14613231_0
14692776_0
14692776_2
14692776_4
14692783_1
14692783_2
14692783_4
14692790_0
14692790_1
14692790_2
14692790_3
14692790_4
14692793_4
14692794_0
14692794_1
14692794_2
14692794_3
14692794_4
14692795_0
14692795_1
14692795_2
14692795_3
14692795_4
14692796_0
14692796_1
14692796_2
14692796_3
14692796_4
14692986_2
14692987_0
14692987_4
14692993_0
14692993_1
14692998_0
14692998_1
14692999_1
14693000_3
14693004_1
14693005_3
14693005_4

For comparison below is a list of similar jobs running at that time that were completed (but maybe the slow-down catched them in a late phase, and they finished despite it. You would have to check the starting times.):
14692776_1
14692783_0
14692789_0
14692793_0
14692793_2
14692793_3
14692797_0
14692797_1
14692797_2
14692797_3
14692797_4
14692987_1

Best regards,
Maciej Falkiewicz

Hi thanks for your feedback. As you said it is probably related with the storage performance. We’ll do the same correction on Yggdrasil as we did on Baobab during the latest maintenance. It seems the performances are fare better on Baobab now.

Thanks for your understanding.

Best

Yann