I’m running a model where I need to get the gradient of the loss function w.r.t my input data so I can use it to update previous networks that are in series. I thought I was calling everything right, but I have encounted a problem where the gradients of my loss function w.r.t input data is zero or very close to it. What am I doing wrong? The gradients exist throughout the network, loss is nonzero, etc.
Here’s the code for the backpropagation. after_graph is the output of a graph neural network, and net2 is my graph.
after_graph2 = Variable(after_graph.clone(), requires_grad=True)
after_net2 = self.net2(after_graph2)
# back prop
loss = self.loss_func(after_net2, y)
# find gradients of net 2
loss.backward(retain_graph = True)
loss_graph_output = after_graph2.grad
Here’s the net:
net2 = nn.Sequential(
nn.Linear(input_size2, hidden_sizes2[0]),
nn.Tanh(),
nn.Linear(hidden_sizes2[0], output_size2),
nn.Tanh()
)
What could be going wrong?