I am trying to optimize over the value z and netG is trained GAN model.
Here is the code. I am getting an error. I want to min the loss between the generated images x and the z. something like this:
maxEpochs = 100
z = nn.Parameter(torch.rand(64,100,1,1), requires_grad=True).to(device)
opt = torch.optim.Adam([z]).to(device)
for e in range(maxEpochs):
g_z = netG(z)
generatedImages = fakeimages.to(device)
loss = nn.MSELoss()(generatedImages, g_z)
# calculate gradient
if e % 100 == 0:
print( '[%d] epoch: %0.5f, Loss: %0.5f' % (e, loss))
I am getting the error
ValueError("can't optimize a non-leaf Tensor")
Can anyone help me out in resolving the error?
Let’s start with the first three lines of the code you posted:
Let me assume that device refers to a gpu.
Your second line of code creates a tensor on the cpu and then
“moves” it to the gpu by creating a gpu copy of the original cpu
tensor. Although the original cpu tensor is a “leaf” tensor, the
copy-to-gpu operation counts as a “computation” so that the
gpu tensor is not a leaf tensor, hence the error you report:
As an aside, your third line of code, as posted, is fully bogus, and
will throw an error, even if you try to construct your Adam optimizer
with a leaf tensor. (In general, a pytorch Optimizer doesn’t have a .to() method.)
>>> opt = torch.optim.Adam([x]).to ('cuda')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Adam' object has no attribute 'to'
Construct the Parameter you wish to optimize directly on the gpu, as
is done with y in the example I gave above, and then use it to construct
the Optimizer (but don’t attempt to “move” the Optimizer):