Hi, I want to make a backward pass that will calculate gradient update need just for one specific tensor.
I understand that the tensors that are closer to the end of the network will also get a gradient calculation, but my goal is just for the specific tensor.
So how can I stop the loss.backword() operation in some layer? The main reason btw is to save time.
As said I’m not trying to escape calculating the layers between the desire tensor and output, but on earlier layers.
One thing that I’m never sure about, does
requires_grad_(False) block the gradient from moving to earlier layers?
lets say I want the gradient just for conv3:
x = self.conv1(x)
x = self.conv2(x)
x = self.conv3(x)
out = self.conv4(x)
Should this be enough:
or this is a must:
The latter. If you picture the autograd tree,
self.conv2.requires_grad_(False) only cuts the bits from the “stem” to the leaves in
self.conv2 (i.e. weight/bias).
Thanks, Is it possible to do the following:
out = model(input_var,first=1)
loss_1 = criterion(out, t)
loss_2 = criterion(out, t)
Means, does the .requires_grad_(True) option need to be submitted in the forward pass of the network, or it can be also before the forward pass
You need it before the forward pass.
So technically, it’s not a Module but a Parameter/Tensor property and you need to decide before you use the Tensor whether you want to differentiate later so that the autograd can start recording.