requires_grad=True/False dynamically

2

Hi,

The property of a layer being “frozen” by setting its parameters .requires_grad to False will impact it only during the forward pass.
The simplest way to do what you want I think is to create two optimizers:

optimizer1 = optim.SGD(Net.conv1.parameters(), ...)
optimizer2 = optim.SGD(Net.conv2.parameters(), ...)

And then your training loop become:

1. compute y = Net(x)
2. loss1 = criterion1(y, target1)
3. optimizer1.zero_grad()
4. loss1.backward(retain_graph=True)
5. optimizer1.step()
6. loss2 = criterion2(y, target2)
7. optimizer2.zero_grad()
8. loss2.backward()
9. optimizer2.step()
3 Likes