So, I am trying to recreate this TensorFlow function in PyTorch, but it doesn’t seem to be working:
d_previous = tf.placeholder("float32") d_logit = fwd_gradients(T("softmax2_pre_activation"), T(layer_name), d_previous) def fwd_gradients(ys, xs, d_xs): """Forward-mode pushforward analogous to the pullback defined by tf.gradients. With tf.gradients, grad_ys is the vector being pulled back, and here d_xs is the vector being pushed forward. By firstname.lastname@example.org from https://github.com/renmengye/tensorflow-forward-ad/issues/2 """ v = tf.zeros_like(ys) g = tf.gradients(ys, xs, grad_ys=v) return tf.gradients(g, v, grad_ys=d_xs)
And my attempt at recreating the function in PyTorch:
def fwd_gradients(ys, xs): v = torch.zeros_like(ys) g = torch.autograd.grad( outputs=[ys], inputs=xs, grad_outputs=[v], retain_graph=True, ) out = torch.autograd.grad( outputs=[g], inputs=v, grad_outputs=torch.zeros_like(xs), retain_graph=True, )
Currently the above PyTorch code results in the following error:
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
How do I get the function working correctly?