I have come across the infamous RuntimeError: Trying to backward through the graph a second time... error while executing the code given below. I have read all the existing posts on the issue and could not resolve it myself. Passing retain_graph=True in backward() fixes the issue in the provided snippet, however, the snippet is only an oversimplified version of a large network where retain_graph=True changes the error to the following:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3000, 512]], which is output 0 of TBackward, is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
I tried setting torch.autograd.set_detect_anomaly(True) and determining the point of failure, but all that I tried failed and the error persisted.
I suspect that if the error in the current snippet is fixed without retain_graph=True then the same solution should work for the larger network.
Therefore, I want to understand why is it that backward() works fine for first two tensors in batched_feats, while fails for the third one. I would really appreciate if someone can help me see the reuse of an intermediate result that has been freed.
# Code Snippet
import numpy as np
import torch
def batch_matrix(vector_pairs, factor=2):
baselen = len(vector_pairs[0]) // factor
split_batch = []
for j in range(factor):
for i in range(factor):
start_j = j * baselen
end_j = (j+1) * baselen if j != factor - 1 else None
start_i = i * baselen
end_i = (i+1) * baselen if i != factor - 1 else None
mini_pairs = vector_pairs[start_j:end_j, start_i:end_i, :]
split_batch.append(mini_pairs)
return split_batch
def concat_matrix(vectors_):
vectors = vectors_.clone()
seq_len, dim_vec = vectors.shape
project_x = vectors.repeat((1, 1, seq_len)).reshape(seq_len, seq_len, dim_vec)
project_y = project_x.permute(1, 0, 2)
matrix = torch.cat((project_x, project_y), dim=-1)
matrix_ = matrix.clone()
return matrix_
if __name__ == "__main__":
vector_list = []
for i in range(10):
vector_list.append(torch.randn((5,), requires_grad=True))
vectors = torch.stack(vector_list, dim=0)
pmatrix = concat_matrix(vectors)
factor = np.ceil(vectors.shape[0]/6).astype(int)
batched_feats = batch_matrix(pmatrix, factor=factor)
for i in batched_feats:
i = i + 5
print(i.shape)
summed = torch.sum(i)
summed.backward()
The code produces the following output and error trace:
torch.Size([5, 5, 10])
torch.Size([5, 5, 10])
Traceback (most recent call last):
File "/home/user/PycharmProjects/project/run.py", line 43, in <module>
summed.backward()
File "/home/user/anaconda3/envs/diff/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/user/anaconda3/envs/diff/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
Variable._execution_engine.run_backward(
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.