Optimize input instead of network weights

Hi,

I am trying to optimize the inputs in some required task and I don’t want to update my network as its freezed. I have written a minimal example but its not working as z has the same value in all iterations. I am sure that I am doing some silly mistake in this process. Any guidance is highly appreciated.
thanks.

``````import torch

z = torch.rand((1,6))
optimizer = torch.optim.SGD([z], lr= 0.1)

criteria = torch.nn.MSELoss()

for i in range(10):
print(z)
loss = criteria(z, z+torch.rand(1))
#print(loss)
loss.backward()
optimizer.step()

##output
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
tensor([[0.1105, 0.8152, 0.2820, 0.1122, 0.6645, 0.7211]], requires_grad=True)
``````

There is no learning happening here. You are adding a constant value to your z and using that as the target value while calculating MSE loss. In that case, the MSE will also be a constant and hence, the gradient of the loss function with respect to z will be 0. You can check the gradient values by printing them after each loop.

``````for i in range(10):
loss = criteria(z, z+torch.rand(1))
#print(loss)
loss.backward()
optimizer.step()
``````

Output:

``````tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
tensor([[0., 0., 0., 0., 0., 0.]])
``````