How can I modify the saved value used for backpropagation?

autograd
1

It may seem weird, but I’m trying to modify the backpropagation itself. (It’s hard to explain in short, but it’s about the adversarial attack.)

I want to modify the saved values of the graph that used in backpropagation. I want something like :

x = Variable(torch.ones(5,3,32,32))
conv = nn.Conv2d(3,24,kernel_size=3, padding=1)
y = F.conv2d(x, conv.weight)
loss = y.sum()
loss.backward()
print(conv.weight.grad[0,0,0]) # tensor([4500., 4500., 4500.])

x = Variable(torch.ones(5,3,32,32))
conv = nn.Conv2d(3,24,kernel_size=3, padding=1)
y = F.conv2d(x, conv.weight)
loss = y.sum()
x.data = 2*x.data
loss.backward()
print(conv.weight.grad[0,0,0]) # It gives ([4500., 4500., 4500.]), but I want ([9000., 9000., 9000.])

It’s not x, also not inside the nn.Conv2d, and F.conv2d looks like a function and looks like have no place for saving something. Where should I found it?

Any help would be greatly appreciated. Thank you.

2

There are a few issues in the code:

  • You shouldn’t manipulate the .data attribute, as Autograd cannot track these changes, which might result in wrong results. If you want to manipulate the data without tracking the computation, use with torch.no_grad instead
  • Manipulating x after the gradient were calculated, doesn’t change the gradients anymore
  • If you would like to scale the gradient, this would work:
x = torch.ones(5,3,32,32)
conv = nn.Conv2d(3,24,kernel_size=3, padding=1)
y = F.conv2d(x, conv.weight)
loss = y.sum() * 2
loss.backward()
print(conv.weight.grad[0,0,0]) 
> tensor([9000., 9000., 9000.])
  • Variable is deprecated since PyTorch 0.4.0, so you can now use tensors :wink:
3

Thank you. But I need to do far more complicated modifications than just scaling the gradient.
So I want to get ‘wrong’ results by deceiving pytorch. Is there no way at all?

4

I wouldn’t recommend to try to “deceive” PyTorch. :wink:
If you explain your use case a bit, we might come up with a clean approach, e.g. by using hooks on the parameters etc.

5

Thank you. I really hope there is such a way.
I have two inputs, x1 and x2. Let’s consider a simple linear layer. The gradient is originally calculated like :

def backward(self, grad_output) : 
    input, weight = self.saved_tensors
    grad_input = grad_output.mm(weight)
    grad_weight = grad_output.t().mm(input)
    return grad_input,  grad_weight

I want to do something like :

def backward(self, grad_output) : 
    input1, weight = self.saved_tensors
    input2, weight = self.saved_tensors2
    grad_input = grad_output.mm(weight1)
    grad_weight = grad_output.t().mm(input1+input2)
    return grad_input,  grad_weight

I can’t just forward with x1+x2, since It changes nonlinear activations like ReLU, MaxPool, etc. Furthermore, grad_output should be calculated with x1, not with x1+x2. This is really unusual situation, and I found that even deceiving pytorch is not simple. I’m totally blocked :tired_face: and would be really happy with any help.

Thank you.

6

you can define custom Linear layer with your .forward and .backward. See https://pytorch.org/docs/stable/notes/extending.html