How can we use autograd for sequence optimization (in a for loop)?

autograd
#1

I want to optimize a sequence in a for loop using Pytorch Autograd.

for i in range(10):
     x = f(x,z[i])

Say the sequence length is 10. I want to optimize x as well as z(z is a tensor array). Note the x will be updated in the loop.

To run this, I have to open:

loss.backward(retain_graph=True)

Is there a better way to do so (To make it run faster)?

#2

I’m not sure why you need to use detect_anomaly, as it’s a debugging tool and is expected to slow down the code. Could you explain the reason a bit more?

#3

Sorry, my mistake.
I am using

loss.backward(retain_graph=True)
#4

Using retain_graph would only be necessary if you need to keep the intermediate tensors alive after a backward operation. I’m still unsure why it’s needed in your case, so would need more information about the actual use case.