Trying to backward through the graph a second time with partially modified intermediate tensor

Hi! I am writing some PyTorch code. I have a Transformer model that predicts the next word. And I also have an adjunc model (e.g. GNN) that outputs some information to be combined in the Transformer. My model looks like this:

class LMGNN(Transformer):
def init(self, LM_params, GNN_params):
self.gnn = GCN(GNN_params)

def forward(self, tokens, graph):
    node_emb.= gnn(graph.x, graph.edge_index)
    (code for passing tokens to Transformer)
    h[0, 0, 0] = h[0, 0, 0,] + node_emb[0, 0] # not the real code, just a simple example
    for layer in self.layers:
        h = layer(h)
    return self.output(h)

As you can see, I try to modify some part of the intermediate parameter h of the Transformer with the output from another model (e.g. GNN). But it comes to a RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). I tried to set the retain_graph=True (I know it is not the solution), then it comes to another error, giving me in_place variable error.

I would appreciate any assistance to this annoying issue.

If it is erroring because a tensor saved for backward is being modified in-place, you can try Automatic differentiation package - torch.autograd — PyTorch 2.1 documentation