RNN loss second-order derivative

I’m trying to take a second-order derivative of the RNN loss function, but I got an error “trying to differentiate twice a function that was markedwith @once_differentiable”. Does this mean the current pytorch implementation does not support 2nd order derivative?

it doesn’t support 2nd order derivative for RNNs yet.

Wow! I have just experienced this limitation. I am implementing an WGAN with an LSTM in the generator and an LSTM in the discriminator and when I calculate the gradients I get

RuntimeError: CudnnRNNLegacyBackward is not differentiable twice

Is there any workaround? When do you think it is going to be implemented? :smiley:

Thanks a lot

You could use SLOGAN. The gradient penalty won’t be the hardest bit.

Best regards


Wow! I didn’t know it! It is really good; impressive work. I have just adapted the SLOGAN loss to my net (for language generation) and now I can train it with 2 RNNs. It seems to be stabilizing, though it gets into a hard mode collapse. Lets wait a bit still… :smiley: Thanks!!!

Any updates on the issue?
because I am running into same problem RuntimeError: CudnnRNNLegacyBackward is not differentiable twice

As far as I know, this is still the same. You might have better luck using LSTMCell and friends.

Best regards


1 Like

Has there been any update on this? I am facing the same problem.