Tensor(-inf, inf, inf....inf) obtained on GPU

Hi Everyone
Recently I came across a bizarre issue.

The output of the convolutional layer is different on the CPU and GPU. CPU gives correct conv output, whereas some values in the output of the conv layer are becoming inf or -inf on GPU

Hi Prakhar!

If you are using an Ampere-architecture gpu (e.g., an A100), then you
might be seeing the effects of the dreaded TensorFloat32 floating-point
format. If so, you could try disabling TF32, as discussed in the linked
documentation, with torch.backends.cudnn.allow_tf32 = False.

(If this is not your issue, please post a simplified, fully-self-contained,
runnable script that reproduces your issue, together with its output and
your gpu model and pytorch and cuda versions.)


K. Frank