I have a problem that might have a very simple solution but it bugs me for a few hours now.
I have several models defined as:
class Model(nn.Module): def __init__(self, base): super(Model, self).__init__() self.base = base self.coeff = nn.Parameter(torch.randn(len(base)), requires_grad=True) def forward(self, x): res =  for x_i in x: res.append(torch.tensor([c*f(x_i) for c, f in zip(self.coeff, self.base)], requires_grad=True).sum()) return torch.stack(res)
I use a custom loss function, defined as:
class RegressionLoss(nn.Module): def __init__(self): super(RegressionLoss, self).__init__() def forward(self, a, p, y_, y, q, w): return (w * a * p * torch.abs(y_ - y) ** q).mean()
All the models and the loss modules sit in seperate lists, i.e. I have 2 lists names models and loss_fn. I also have a list of LBFGS optimizers, named optimizers.
I call the following function in a loop:
def lbfs_KHM(self, k, a_values, p_values, x, y, w, q=2): def closure(): self.optimizers[k].zero_grad() output = self.models[k](x) loss = self.loss_fn[k](a_values, p_values, output, y, q, w) loss.backward() return loss self.optimizers[k].step(closure)
Altough I assumed this will create different graphs for different models and losses, I get the following error when calling loss.backward:
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.