I am training a segmentation model using unet. In the decoder part of network, The input to a Conv2d layer (which is self.conv[0] here) is x (the extracted features from a batch of imgs), and it doesn’t have nans or infs. I am not using bias in the Conv2d and none of the weights (or grads) in it is inf or nan. But I am still getting infs in the output of Conv2d layer. How is this possible?
I am attaching the issue screenshot (from VS Code debug mode).
Also the screenshot of the relevant part of model class is below:
Is there something I am missing? This feels not specific to this problem or network, but to the Conv2D operation itself.
You might be running into an overflow during the conv calculation. Could you check the min/max/mean stats of the weights as well as the inputs?
Yeah, the max/min of input is within ±1000, and the weights are all between -1/1.
I don’t know how the channels are set, but given these ranges I wouldn’t expect to see any overflows for channel dims <=1024. Could you share a minimal and executable code snippet which would reproduce the issue or any checkpoint?