RNN loss second-order derivative

xiaofan · September 20, 2017, 6:36pm

I’m trying to take a second-order derivative of the RNN loss function, but I got an error “trying to differentiate twice a function that was markedwith @once_differentiable”. Does this mean the current pytorch implementation does not support 2nd order derivative?

smth · September 21, 2017, 2:51am

it doesn’t support 2nd order derivative for RNNs yet.

ivallesp · January 26, 2018, 2:44pm

Wow! I have just experienced this limitation. I am implementing an WGAN with an LSTM in the generator and an LSTM in the discriminator and when I calculate the gradients I get

RuntimeError: CudnnRNNLegacyBackward is not differentiable twice

Is there any workaround? When do you think it is going to be implemented?

Thanks a lot

tom · January 26, 2018, 3:50pm

You could use SLOGAN. The gradient penalty won’t be the hardest bit.

Best regards

Thomas

ivallesp · January 26, 2018, 7:29pm

Wow! I didn’t know it! It is really good; impressive work. I have just adapted the SLOGAN loss to my net (for language generation) and now I can train it with 2 RNNs. It seems to be stabilizing, though it gets into a hard mode collapse. Lets wait a bit still… Thanks!!!

tanyaisanumber · March 4, 2018, 2:49pm

Any updates on the issue?
because I am running into same problem RuntimeError: CudnnRNNLegacyBackward is not differentiable twice

tom · March 4, 2018, 5:52pm

As far as I know, this is still the same. You might have better luck using LSTMCell and friends.

Best regards

Thomas

Sachin_Kumar · May 1, 2019, 10:07pm

Has there been any update on this? I am facing the same problem.