Update pair of images with loss function

Hello,

I have a system of face recognition that when I send to that system pair of two images, it answers me if it’s the same person. what I try to do is to change a little bit of the images with the answer from the loss function to change the response of the system. (Adverdrial Examples)

The two images of type ndarray and their shape are: (1, 3, 112, 112) with float32 as dtype.
I cast them to torch type and put them on Adam optimizer (also enable the requires_grad=True).

t_image1 = torch.from_numpy(image1).to(device)
t_image2 = torch.from_numpy(image2).to(device)
t_image1.requires_grad = True
t_image2.requires_grad = True
optimizer_image = torch.optim.Adam(params = [t_image1, t_image2], lr=0.0001)

The loss function that I selected is BinaryCrossEntropy.

loss = losses.getLossFunction("binarycrossentropy")

Now process that I query the system twice and try to change the label with the loss:
The predict that return from system is 1 or 0 (int).

fr1_predict = fr1.predict(t_image1, t_image2)
fr2_predict = fr2.predict(t_image1, t_image2)

Now doing the loop:
I save the images before the process to make a cosine similarity to see the change.

while fr1_predict != 1 or fr2_predict != 0:
        prev_image1 = t_image1
        prev_image2 = t_image2

        l = loss(torch.tensor(float(fr1_predict), requires_grad=True), torch.tensor(float(fr2_predict), 
                     requires_grad=True))
        print("loss:", l)
        optimizer_image.zero_grad()
        l.backward()
        optimizer_image.step()

        fr1_predict = fr1.predict(t_image1, t_image2)
        fr2_predict = fr2.predict(t_image1, t_image2)
        print("fr1_predict:", fr1_predict)
        print("fr2_predict:", fr2_predict)
        print("fr1_predict == fr2_predict:", fr1_predict == fr2_predict)
        print("cosine similarity image 1:", cosine_similarity(torch.flatten(t_image1), torch.flatten(prev_image1), dim= 0))
        print("cosine similarity image 2:", cosine_similarity(torch.flatten(t_image2), torch.flatten(prev_image2), dim= 0))


The print of the loss and cosine are, that prints returns a lot of time with a same result in a series:

loss: tensor(0., grad_fn=<BinaryCrossEntropyBackward0>)
fr1_predict: 1
fr2_predict: 1
fr1_predict == fr2_predict: True
cosine similarity image 1: tensor(1., grad_fn=<DivBackward0>)
cosine similarity image 2: tensor(1., grad_fn=<DivBackward0>)

Why the photos don’t change in that process? What do I miss?

Thank you!