Loss.item() is inf or nan

1

I defined a new loss module and used it to train my own model. However, the first batch’s loss always get inf or nan, which leads to fail.

I try to print the loss item info as follows:

loss item: inf
loss item: 7.118189334869385
…
loss item: 7.123733997344971

what may it happpens? I test the loss module and it works with some synthesis data. And the module is implemented with torch functions as well.

Could someone occcur this kind of problem?

2

Could you check your input for NaN or Inf values by calling torch.isnan and torch.isinf on it?
If that’s not the case, you could use the anomaly detection util to debug your model.

3

Thanks for your suggestion and I have solved this problem.

It is because of the torch.exp function and sometimes it will take a large input and torch.exp will get inf value.

4

Can you elaborate more on this?

1 Like