I test the backward of grid sample function and find that the gradient of input is non-deterministic even with cudnn.deterministic = True
or cudnn.enabled = False
:
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.backends import cudnn
cudnn.deterministic = True
l1_loss = nn.L1Loss()
def grid_sample_test(seed):
torch.manual_seed(seed)
grid = torch.randn(32, 32, 32, 2).cuda().requires_grad_()
inputs = torch.randn(32, 3, 32, 32).cuda().requires_grad_()
outputs = F.grid_sample(inputs, grid, mode='bilinear', padding_mode='zeros')
targets = torch.randn(32, 3, 32, 32).cuda()
loss = l1_loss(outputs, targets)
print(loss.item())
loss.backward()
return inputs.grad, grid.grad
# cudnn.enabled = False
for i in range(10):
input_grad_1, grid_grad_1 = grid_sample_test(i)
input_grad_2, grid_grad_2 = grid_sample_test(i)
print(i, torch.equal(input_grad_1, input_grad_2), torch.equal(grid_grad_1, grid_grad_2))
Based on my understanding, the backward of F.grid_sample
is non-deterministic when cudnn is enabled since cudnnSpatialTfSamplerBackward
is non-deterministic according to the nvidia cudnn docs.
But why is it still non-deterministic even disabling cudnn?
However, the gradient of grid is deterministic in both cases.
I really hope that I can obtain the deterministic behavior from F.grid_sample
. Needing help~