Hi! I am writing some PyTorch code. I have a Transformer model that predicts the next word. And I also have an adjunc model (e.g. GNN) that outputs some information to be combined in the Transformer. My model looks like this:
class LMGNN(Transformer):
def init(self, LM_params, GNN_params):
super().init(LM_params)
self.gnn = GCN(GNN_params)
def forward(self, tokens, graph):
node_emb.= gnn(graph.x, graph.edge_index)
(code for passing tokens to Transformer)
...
...
...
h[0, 0, 0] = h[0, 0, 0,] + node_emb[0, 0] # not the real code, just a simple example
for layer in self.layers:
h = layer(h)
...
...
return self.output(h)
As you can see, I try to modify some part of the intermediate parameter h of the Transformer with the output from another model (e.g. GNN). But it comes to a RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). I tried to set the retain_graph=True (I know it is not the solution), then it comes to another error, giving me in_place variable error.
I would appreciate any assistance to this annoying issue.