RuntimeError: cudnn RNN backward can only be called in training mode during backpropagation

autograd
1

Hi. I am trying to perform an adversarial attack on a model, where I have to calculate gradients of loss over inputs, something like this:

model.eval()
for prediction != target:
     prediction = model(input)
     loss = criterion(prediction, target)
     loss.backward()
     input = input - k * (input.grad())

But, following error is encountered Pytorch cudnn RNN backward can only be called in training mode. I tried the solutions posted here, which are:

  1. Call .train() on the rnn module after using model.eval(). Used this as follows:
model.eval()
for prediction != target:
     prediction = model(input)
     loss = criterion(prediction, target)
     model.train()
     loss.backward()
     input = input - k * (input.grad())
     model.eval()

It doesn’t help. The error is still there.

  1. call .eval() only on the necessary modules: I have 2 doubts regarding this solution.
    a. My model under consideration is quite big, it is advisable to go for this?
    b. Does it affect my prediction accuracy in any way?

  2. disable cudnn via torch.backends.cudnn.enabled = False. Going for this solution, I think I won’t be able to GPU. Without the GPU the running times is 4-5 times to that of time taken for execution with the GPU.

Can you please suggest me workaround/solution? Thanks in advance.

2

Could you post a (minimal) model definition including all necessary shapes and arguments to reproduce this issue?

  1. The predictions shouldn’t be affected in any way.

  2. The GPU will still be used, if you disable cudnn. As the call suggests cudnn won’t be used and the native CUDA implementations will be called instead.

1 Like