Setting part of the training value into zero

2

Hi,

This problem raises in two kind of situations:

  1. When you use inplace=True in your code and somehow regarding your model, backpropagation needs the values unchanged.
  2. When you change a value like the image you have attached and backprop needs previously value again to compute gradient.

In the first situation if cannot change the way you have implemented your model, you can just set inplace=False and it will be ok.

But in the second one, you have to preserve let assume X value.

Here is an small example you can test:

A = torch.nn.Parameter(torch.randn(3, 3))
    B = torch.randn(2, 4, 3)
    B[0, :, :] = B[0, :, :].mm(A)
    loss = B.sum()
    loss.backward()

This will give you same error you have posted. And here is the solution:

A = torch.nn.Parameter(torch.randn(3, 3))
B = torch.randn(2, 4, 3)
z = B[0, :, :].mm(A)
loss = z.sum()
loss.backward()

As you can see, in the second example, B matrix, in your case X has not been changed so gradient could be computed.

By the way, operations like x += y, x[:, ...] = y etc will be considered inplace operation.

Good luck

2 Likes