RuntimeError: hook 'backward' has changed the size of value

1

I’m trying to modify grad_input in my backwards hook function, but I am getting the following error:

Traceback (most recent call last):
  File "st.py", line 260, in <module>
    optimizer.step(feval)
  File "/usr/local/lib/python2.7/dist-packages/torch/optim/lbfgs.py", line 101, in step
    orig_loss = closure()
  File "st.py", line 255, in feval
    loss.backward()
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/variable.py", line 167, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/__init__.py", line 99, in backward
    variables, grad_variables, retain_graph)
RuntimeError: hook 'backward' has changed the size of value

    def backward(self, grad_input, grad_output):

        gradInputTuple = ()
        gradOutput = grad_output[0] 
        for input in grad_input:
          print(str(input.size()) + " " + str(self.target.size()))
          if input.nelement() == self.target.nelement():
             self.gradInput = mse_loss2(input, self.target)
          if self.normalize == 'True':
             self.gradInput = self.gradInput.div(torch.norm(self.gradInput, 1) + 1e-8) # Normalize Gradients
          self.gradInput = self.gradInput * self.strength
          self.gradInput = self.gradInput + gradOutput
          print("gradInput.size():")
          print(str(self.gradInput.size()))
          gradInputTuple = list(gradInputTuple)
          gradInputTuple.append(self.gradInput)
          gradInputTuple = tuple(gradInputTuple)
        else:
          self.target = gradOutput
        return gradInputTuple #self.gradInput

I am trying to replicate this Lua/Torch code:

function ContentLoss:updateGradInput(input, gradOutput)
  if self.mode == 'loss' then
    if input:nElement() == self.target:nElement() then
      self.gradInput = self.crit:backward(input, self.target)
    end
    if self.normalize then
      self.gradInput:div(torch.norm(self.gradInput, 1) + 1e-8)
    end
    self.gradInput:mul(self.strength)
    self.gradInput:add(gradOutput)
  else
    self.gradInput:resizeAs(gradOutput):copy(gradOutput)
  end
  return self.gradInput
end

What am I doing wrong here?

2

I do have the same error message, but for a slightly different problem. In the end, did you fix it?

In my case, I would like opt.step() to use complete different gradient values.
‘new_gradients’ is a dictionary for which the keys() have the same name as that the model parameters. For example,

block.fc1.weight
block.fc1.bias
etc…

Thus, we have that

	f = model(x)

	loss = criterion(f, y)
	
	for (key, w) in model.named_parameters():
		w.register_hook(lambda grad: new_gradients[key])

	optimizer.zero_grad()
	loss.backward()

There is an error at the very last line,

RuntimeError: hook ‘lambda’ has changed the size of value

Help would be appreciated.

3

Hi,

Your error comes from the fact that one of your lambda returns a tensor which is not of the same size as what it got as input. You can replace the lambda by a real function and add some extra prints. In this case, I think the batch size is handled in a way you don’t expect: the gradient of the hook for a tensor of size HxW will be BxHxW and is then summed over the batch dimension if I remember correctly.

For your use case, if you just want to set the gradients, you don’t actually need to run a backward pass:

optimizer.zero_grad()
with torch.no_grad():
  for (key, w) in model.named_parameters():
    w.grad.copy_(new_gradients[key])

optimizer.step()
1 Like
4

Hi @ProGamerGov, sorry to bring this up. I ran into the same RuntimeError: hook has changed the size which makes me confused about the grad_input tuples. Correct me if I am wrong, the tuple should be (act_grad, weight_grad, bias_grad) right?
So what was wrong with your previous code? Did you solve it? Thank you very much.

5

I never solved that particular issue, but I would recommend using a custom autograd function if you want to change things in the backward pass: https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html