Gpu008 Faulty GPU 3

HPC Support HPC issues
1

Hello,

On bamboo, the node gpu008 has a faulty GPU (id 3).

Here is the error message encountered when trying to use it:

CUDA error: uncorrectable ECC error encountered
Search for `cudaErrorECCUncorrectable' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.

Thank you

2

Hello @Raphael.Rubino

The node has been drained from production.

Could you share the steps to reproduce the issue so that we can identify the root cause more precisely?

3

Sure @Adrien.Albert,

Please log on the node gpu008 with GPU #3 allocated (I have tested with single GPU instances only).

Then, in a python console (I have tested with python 3.10):

import torch
# declare a tensor
t=torch.zeros((2,2))
# print tensor
print(t)
# printed result
> tensor([[0., 0.], [0., 0.]])
# move the tensor to GPU
t=t.to("cuda")

The expected result should be:

# print the tensor once moved to GPU
print(t)
# printed result
> tensor([[0., 0.],
        [0., 0.]], device='cuda:0')

Instead, we get the error reported above in this thread.

Thank you.

4

Hi,
I also got same error.

5

Hello

The node has been rebooted and seem working now