Baobab: "No devices were found" on one of the GPU002

[Malte.Algren]

Hi HPC,

I allocated some resources on gpu002 and got a gpu. However when running code or nvidia-smi there is no device found (see picture).

Best,
Malte

[Raphael.Rubino]

Hello,

Yes I think it is GPU#3 (index starts at 0) on baobab:gpu002 which is down.
I created a report a while ago:

Error when running GPU jobs on gpu002 (GPU #3 seems down)

Primary informations Username: rubinor Cluster: baobab Node: gpu002 Description Error when running GPU jobs on gpu002. Error messages: – Python RuntimeError: No CUDA GPUs are available – When the job starts, quick ssh to gpu002 and run nvidia-smi No devices were found Current usage of gpu002: CfgTRES=cpu=10,mem=257000M,billing=10,gres/gpu=6,gres/gpu:titan=6 AllocTRES=cpu=8,mem=94G,gres/gpu=3,gres/gpu:titan=3 Steps to Reproduce The error only happens if gpu002 is already in use. To rep…

[Yann.Sagon]

Indeed, @Raphael.Rubino I’ve answered your previous post right now. And @Malte.Algren we’ll check the node, thanks.

Best

[Yann.Sagon]

This is fixed, the node as 6 GPUs again.
Best