I’ve found this one example which illustrates my problem:
import torch
from torch.autograd import grad
import torch.nn as nn
# Create some dummy data.
x = torch.ones(2, 2, requires_grad=True)
gt = torch.ones_like(x) * 16 - 0.5 # "ground-truths"
# We will use MSELoss as an example.
loss_fn = nn.MSELoss()
# Do some computations.
v = x + 2
y = v ** 2
# Compute loss.
loss = loss_fn(y, gt)
print(f'Loss: {loss}')
# Now compute gradients:
d_loss_dx = grad(outputs=loss, inputs=x)
d_loss_dv = grad(outputs=loss, inputs=v)
print(f'dloss/dx:\n {d_loss_dx}')
With this approach, I can calculate dloss / dx using the grad function, but when the function is called a second time to calculate dloss / dv (which we just assume, I need the result for) I get the Runtime Errror:
RuntimeError: Trying to backward through the graph a second time (or directly access saved variables after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved variables after calling backward.
Probably due to the fact that backpropagation was already called once, and we would not want to do that again, I guess?
So my question is, is there a better way to do what I try to achieve above, calculating dloss_dx and d_loss_dv?
Thanks in advance!