Different behaviour for jacwd and grad. For grad, RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 1]], which is output 0 of TanhBackward

Hi, I’m getting this error and I know I have in-place operations in my code. However, the wired thing is that “jacfwd” works, but grad does not. I don’t know why this is happening. Any help would be appreciated. Next is an simplified example.

 import torch
 from torch import nn
 import numpy as np
 from torch.func import vmap
 from torch.func import jacfwd,grad
 def test(cart):
     sph=cart.new_ones(10,cart.shape[0])
     for i in range(1,10):
          sph[i]=cart[i]*sph[i-1]
     return torch.sum(sph)
 
 cart=torch.randn((10))
 jac=jacfwd(test)
 print(jac(cart))
 gradient=grad(test)
 print(gradient(cart)) 

Hi yl!

jacfwd() use forward-mode auto-differentiation, so, unlike grad(), it doesn’t
have inplace-modification issues. (Just for fun, you might try jacrev(), which
uses “conventional” backward-mode auto-differentiation.)

Best.

K. Frank

Thank you for your answer!