I test the backward of grid sample function and find that the gradient of input is non-deterministic even with
cudnn.deterministic = True or
cudnn.enabled = False:
import torch import torch.nn as nn import torch.nn.functional as F from torch.backends import cudnn cudnn.deterministic = True l1_loss = nn.L1Loss() def grid_sample_test(seed): torch.manual_seed(seed) grid = torch.randn(32, 32, 32, 2).cuda().requires_grad_() inputs = torch.randn(32, 3, 32, 32).cuda().requires_grad_() outputs = F.grid_sample(inputs, grid, mode='bilinear', padding_mode='zeros') targets = torch.randn(32, 3, 32, 32).cuda() loss = l1_loss(outputs, targets) print(loss.item()) loss.backward() return inputs.grad, grid.grad # cudnn.enabled = False for i in range(10): input_grad_1, grid_grad_1 = grid_sample_test(i) input_grad_2, grid_grad_2 = grid_sample_test(i) print(i, torch.equal(input_grad_1, input_grad_2), torch.equal(grid_grad_1, grid_grad_2))
Based on my understanding, the backward of
F.grid_sample is non-deterministic when cudnn is enabled since
cudnnSpatialTfSamplerBackward is non-deterministic according to the nvidia cudnn docs.
But why is it still non-deterministic even disabling cudnn?
However, the gradient of grid is deterministic in both cases.
I really hope that I can obtain the deterministic behavior from
F.grid_sample. Needing help~