running this code:
def do_something(self, img): # Create mask with weak edges set to 1, else 0 masked_img = img.clone() masked_img[torch.logical_not(img == 0.5)] = 0 masked_img[img == 0.5] = 1 # Calculate weak edges that are changed to strong edges changed_edges = img.clone() changed_edges[:] = 0 changed_edges[((self.conv4(img) > 1) & (masked_img == 1))] = 1 # Add changed edges to already good edges changed_edges[img == 1] = 1 return changed_edges
RuntimeError: Function ‘PowBackward0’ returned nan values in its 0th output.
And with anomaly detection set to false I can see that my kernel weights have turned to NaN’s.
If I delete the line “changed_edges[:] = 0” the network trains without problems. I’ve tried some different methods to cope with this problem but I am out of ideas here. Seems like the gradients become extremely big so that my network gets rekt during training? Help or some hints would be highly appreciated. Thanks.