CUDA error: device-side assert triggered in optimizer.step()

1

I am trying to train on the unet from torch.hub. Some of the image-label pairs give me a CUDA error: device-side assert triggered.

Screenshot%20from%202019-12-11%2000-52-03
Screenshot%20from%202019-12-11%2000-52-03806×341 40.9 KB

How Can I fix this?

2

Does this model work on the CPU without any errors?
Usually you might get a better error message by running it on the CPU.

3

I tried running it on the cpu. I was able to fix the problem with the error message.