How can i update an existing variable across different inferences in a computational graph (a memory variable that gets updated for every input during a forward pass) making sure that the gradients are computed correctly for that variable?
For example, I want to do self.x = self.x + input every inference, where self.x is an internal variable for a layer.
I get an error that says the following:
RuntimeError: Trying to backward through the graph a second time (or directly access
saved tensors after they have already been freed). Saved intermediate values of the
graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=
True if you need to backward through the graph a second time or if you need to acces
s saved tensors after calling backward.
Upon setting retain_graph to True, I get the following error:
RuntimeError: one of the variables needed for gradient computation has been modified
by an inplace operation: [torch.cuda.FloatTensor [70, 32, 4, 4]] is at version 2; e
xpected version 1 instead. Hint: enable anomaly detection to find the operation that
failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
As a workaround, is it functionally equivalent if I detach() and reassign value to the same variable?