@ptrblck Hello, I have checked for invalid values of input, masks, model output, and loss too. There are no nan or inf values in any of them. One weird thing is I am finding the nan values after some epochs and not in some of the initial epochs. What could be the reason behind this?. Thank you.
How does the loss look during the training? Is it decreasing smoothly or are you seeing any high values?
@ptrblck Thanks for the reply. Yes, loss is decreasing smoothly for both the training and validation losses. There are no high values or sudden peaks in it.
If no inputs are invalid and your model suddenly outputs NaNs without a particular high loss values in the previous iterations, some parameters might have been pushed to invalid values.
Could you check all parameters in each iteration for NaNs and Infs via:
for name, param in model.named_parameters():
if torch.isnan(param).any() or torch.isinf(param).any():
print('INVALID param in ', name)
Yes. I have tried for training with checking the model parameters. Somehow, this time network got trained without any NaNs in training and validation errors. Thank you.