My GPU is dead while using Nvidia Apex

[ptrblck]

@keunwoo

I’m not sure it is relevant, but there were some weird incidents while using Apex.
When I terminated a python script with ctrl-c, sometimes Apex did not fully terminate and some sub-processes existed. Those sub-processes kept held GPU memories, so I had to kill them manually. These incidents made me suspect Apex.

I’m not sure, if this is caused by apex or PyTorch, as I’ve seen this behavior using plain PyTorch. If I’m not mistaken, this should be fixed in the latest stable release.

@keunwoo

I agree that it might be a hardware problem, but I wonder if a high GPU utilization might harm a gpu.
Is it possible that utilizing GPU 99% for a long time can affect the hardware (e.g. overheat)?

If you didn’t overclocked the GPU, it should be fine. In case your device overheats, e.g. if your GPUs are packed tightly into the case, it should reduce its clock and shutdown as the last step.

@keunwoo

Moreover, is there any other way to utilize gpu up to 90% instead of using Apex?
When I tested some codes, gpu utilization was around 50~70% and I’m not sure whether it is normal.
I wanted to increase it, so I ended up with Apex.

It depends on your code and e.g. you might have a data loading bottleneck.
This post explains some workarounds.