I am slightly confused by the shape of the gradient after the backward pass on a VGG16 Network.
The input dimension of the tensor is [1, 3, 224, 224] , but on backward pass the gradient is [1 , 64, 224, 224] , which corresponds to the output after the first convolution layer in the network.
Shouldn’t the gradient be the same as the size of the input?
How did you check the gradient shape? Could you post the code snippet you were using?
If you want to create gradients for the input, you would have to set the requires_grad attribute of the input tensor to True and could check the gradient via print(input.grad) after the backward call.
The gradient would have the same shape as the input in this case.
self.gradients isn’t defined in the code snippet so I guess you have used some hooks to create it?
def generate_gradients(self, input_image, target_class):
import pdb; pdb.set_trace()
input_image = input_image.cuda()
input_image.requires_grad = True
model_output = self.model(input_image)
# Zero gradients
self.model.zero_grad()
# Target for backprop
one_hot_output = torch.FloatTensor(1, model_output.size()[-1]).zero_()
one_hot_output[0][target_class] = 1
one_hot_output = one_hot_output.cuda()
# Backward pass
model_output.backward(gradient=one_hot_output, retain_graph = True)
# Convert Pytorch variable to numpy array
# [0] to get rid of the first channel (1,3,224,224)
gradients_as_arr = self.gradients.data.detach().cpu().numpy()[0]
return gradients_as_arr
Yes, I am using hooks to create it. When I try to set requires_grad = True, I get RuntimeError: you can only change requires_grad flags of leaf variables. The code is a bit large to put here so I am not able to put a completely reproducible example. My intention is to do guided back-propagation, i believe for this the gradient needs to flow all the back to the inputs so I can get the gradient in the shape of the input (correct me if I am wrong), do I need to set required_grad = True at the time of training then?
The error message would be raised, if input_image is not a leaf variable, i.e. if it was created by another operation. If that’s the case, it might already require gradients and you should be able to access them via imput_image.grad.
Since the current self.gradients shape is different than the input_image.shape, it seems you are registering the hook on a different parameter or layer.
I tried some little more debugging, now input_image.grad returns a [1, 3, 224, 224] tensor as expected and does not throw an error about the input not having grads, however self.gradients.data is still [1, 64, 224, 224].
def hook_layers(self):
def hook_function(module, grad_in, grad_out):
self.gradients = grad_in[0]
first_layer = list(self.model.features_1._modules.items())[0][1][0]
first_layer.register_backward_hook(hook_function)
This is how I am hooking into the layers, as you can see the hook is at the first layer itself, so ideally speaking the shape flowing into the first layer should be equal to [1, 3, 224, 224] and not [1, 64, 224, 224] right?
I would recommend not to use register_backward_hook, as it might fail as described in the docs.
Use parameter.register_hook or grab the .grad attribute directly.
Got it, but will register_backward_hook affect the grad_in of the module? I am correctly placing the hook in the first layer where it should be, the gradient flowing in should be the shape of the image [1, 3, 224, 224] but both grad_in and grad_out are [1, 64, 224, 224] any tips for this?
Since it’s a knows issue, as described in the docs, my recommendation would be to use register_hook instead.
Thanks for your reply. Alternatively, since you said I can directly grab the gradient, I can also assign self.gradient in my code to the gradient of the input itself?
Also thanks a lot for your patience!
Yes, that should be possible.
E.g. you could try to use:
self.gradient = input.grad
Use input.grad.clone() if you want to assign a copy to self.gradient.