How to freeze a single weight of a conv2d layer

vision
2

Zeroing out the gradients should work for optimizers without running estimates.
If you are using an optimizer with internal states (e.g. Adam), this approach will still work, if you zero out the gradients from the beginning:

conv = nn.Conv2d(1, 1 ,3)
weight_reference = conv.weight.clone()
optimizer = torch.optim.Adam(conv.parameters(), lr=1.)

x = torch.randn(1, 1, 10, 10)
out = conv(x)
out.mean().backward()

conv.weight.grad[:, :, 1, 1] = 0.
optimizer.step()

print(weight_reference == conv.weight)
> tensor([[[[False, False, False],
          [False,  True, False],
          [False, False, False]]]])

However, after a valid update was performed, the running averages might update the parameter further.

2 Likes