Cnn convolution layer return nans

1

hi,
i have a cnn model,
the first convolutional layer returns nan values:

x tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], device='cuda:0',
       grad_fn=<SoftmaxBackward>)

it is not happening immidiatly, only after some batches
i have bo idea why

2

Check the input data contains no NaNs, NaNs in convolutions usually come from backprop of NaNs.
Are you performing any risky operation?

3

i have a costume loss, mabey this is the reason?

4

Custom losses tend to be way less stable :slight_smile:
But just check you are not passing negative values to a log, doing anything/0 these kind of things.
Also have a look at
https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly

Still recommend you to check the input data if you apply any more suspicious transform. (Realize normalization of a signal whose values are close to 0 leads to a 0-division for example)

5

my inpet data is only ones and zeros
and my forward is:

 def forward(self, x):
        x = self.dropout_input(x)
        x = x.transpose(1, 2)
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x = self.conv4(x)
        x = self.conv5(x)
        # x = self.conv6(x)
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        x = self.softmax(x)
        return x
6

i only have a sequencial layer, and the data is ones and zeros

7

Soo have a look at your custom loss because there is no way you to find the issue with those layers.