Loss: nan in GPU ubuntu, Valid in Colab

1

Hi Guys,
I notebook file that is working in Colab, and the loss is valid, but when I execute it in Ubuntu with a local GPU, I get some accuracy, but I get nan, as Loss. I have no idea why. Have you seen anything like this?

2

Hard to tell without seeing some code. Maybe start by checking CUDA/CUDNN versions on both platforms?

Colab for me shows:

image
image1205×348 23.7 KB

I have also had problems with old hardware (780ti) a long time ago behaving strangely when memory consumption was close to 100%.

1 Like
3

I have the same problem. did you manage to solve it?