I have trained a Generator named netG (which is fixed in the following code). Given x, then I tried use adam optimizer to optimize the following function to get the z:
def __init__(self, nz):
self.z = nn.Parameter(torch.zeros(1, 10, 1, 1),requires_grad=True)
self.criterion = nn.BCELoss()
def forward(self,x,netG, netD):
l_rec = torch.norm(x-netG(self.z))
The optimization process is:
#Number of repetitions
epoch = 1000
loss_epoch = np.zeros(epoch)
for i in range(epoch):
#Gradient initialization used in optimizer
lss = model(x, netG, netD)
loss_epoch[i] = lss.item()
But the result shows my z doesn’t update (loss not decreasing), the loss is showing in the following figure
Can anyone help me?
Can you try printing
forward(), to see if it becomes non-zero?
Is this expected behaviour? If this tensor stays at all zeroes, will your loss decrease? Or does your loss decreasing depend on this tensor changing?
No, I am expecting the this z should not be zero all the time. My loss decreasing does depend on this tensor. That is why I am expecting the loss to decrease.
In that case you have to write code in your
netG class to ensure that gradients on
z are propagated (I am not sure if this is the correct term; what I mean is that you should tell PyTorch how gradients with respect to
z change, when you call
This is not something that I have done (yet!), so I am not entirely clear on how to do this. I think you need to implement a
backward function inside your
netG class. The code in this message may help you get started.
Thanks! I tried to initialize
torch.rand(1, 10, 1, 1) and then it works. But don’t know why.