Loss showing NaN in 2nd epochs

1

I was training a model with images and loss Binary crossentropy.
The loss seems to be genuine in 1st epoch. But in 2nd epoch after some amount of batches only it will raise nan loss but the accuracy increases. also i am using amp.

loss: 6.89388 ; accuracy: 0.72175: 100%|██████████| 3010/3010 [1:19:28<00:00,  1.58s/it]
loss: 0.56813 accuracy: 0.84400: 100%|██████████| 751/751 [05:47<00:00,  2.16it/s]

loss: nan ; accuracy: 0.84616: 100%|██████████| 3010/3010 [1:13:44<00:00,  1.47s/it]
loss: nan accuracy: 0.84765: 100%|██████████| 751/751 [05:46<00:00,  2.17it/s]

and when calculating AUC with 2nd epoch validation data
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

2

There could be many reasons why this could be the case,
I will suggest to turn on the flag torch.autograd.set_detect_anomaly(True)
This will drop in error wherever the autograd engine encounters NaN and give you the function where the error occurred, It could be advantageous to debug and see what is going on!

3

I found what the error was.
It was caused due to batch normalization

4

can you explain in detail ？ I have the same problem. thanks