Optimizing Part of the Input

Hi all.
I have a trained model, and I am trying to optimize the input (s.t. loss(output) will decrease). I saw some similar topics, but none of them solved my problem.

I have the following flow:


Where A is the only image I want to optimize.

My code:

        model.eval()
        Y = Y.cuda()

        A = torch.tensor(A, requires_grad=True)
        B = torch.tensor(B, requires_grad=False)

        data_optimizer = torch.optim.Adam([A], lr=1.0)

        C = torch.cat((A, B), 1)
        C = torch.tensor(C, requires_grad=True).cuda()

        for i in range(0, iter):

            Y_pred = model(C)

            loss = myloss(Y_pred, Y)

            print('loss: '+str(loss.item()))
            print('random_val: '+str(A[0, 0, 500, 500]))

            data_optimizer.zero_grad()
            loss.backward()
            data_optimizer.step()

I print the loss (which is just noisy) and a random pixel from image (which doesn’t change).
My question is why A doesn’t change in this process?

When I pass C to the optimizer (which is cat of A and B), the prints show that A changes, but B changes as well (which is not acceptable).

Any advice?
Thank you.

Hi,

try this:

A = torch.tensor(A).cuda().requires_grad_(True)  # or torch.tensor(A, requires_grad_=True, device=device) with device="cuda:0" as example
B = torch.tensor(B).cuda()

data_optimizer = torch.optim.Adam([A], lr=1e-3)

C = torch.cat((A, B), 1)

This way we say that only for A the gradients should be calculated.
If you do this torch.tensor(C, requires_grad=True).cuda(), the gradients will actually only flow back to the C tensor, because the graph for the creation of C gets lost (someone can probably explain it better).

2 Likes

Thank you for your reply. It worked nicely.
Another problem was that C = torch.cat((A, B), 1) had to be inside the for loop (since I am iteratively adjusting one of its components).
Thanks!