I am defining my own loss function, and my own loss function has a hyper-parameter Lambda. For example, if the prediction is y, then I define the loss function as Loss = Lambda * y. I want to update my Lambda at some iteration using the current round’s Loss. For example, at some specific iteration, I want my Lambda to be updated as Lambda = Lambda + Loss, then it returns the error of
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
Specifically, my naive code is as follows:
import numpy as np import torch import torch.nn as nn import torch.nn.functional as F import torch.optim as optim batch_size, input, output = 10, 3, 3 model = nn.Linear(input, output) optimizer = optim.Adam(model.parameters(), lr=1e-3) lam = torch.from_numpy(np.array([0.1, 0.1, 0.1])) lam.requires_grad = False for i in range(10): x = torch.rand(batch_size, input) output = model(x) loss = torch.sum(lam*output) if i == 5: lam = lam + torch.clone(loss) optimizer.zero_grad() loss.backward() optimizer.step() print(loss)
I had the feeling that the error was caused by using Loss to update my Lambda. So I used the code
torch.clone(loss), hoping not to influence
loss, it didn’t help. Does anyone know how to fix the problem? Some explanation about why this error occurs would be great!