3 ways of calculating gradient give 3 differents results

autograd
1

Hello,
To make a kind of fast gradient sign attack, I try to calculate the gradient of an image against a softmax loss. The problem is that the image is normalized before entering the model, and I want to get the gradient relative to the unnormalized (untouched) image.

I think the theoretical gradient of the normalization transform:
torchvision.transforms.Normalize(mean=mean, std=std)
can be obtained with:
torchvision.transforms.Normalize(mean=0, std=std)

I tried to calculate the gradient in three different ways but I get 3 different results. What am I doing wrong?

The model I use is a model transferred from AlexNet :

  n_classes = 10
  model = torchvision.models.alexnet(pretrained=False)
  model.classifier[6] = torch.nn.Linear(model.classifier[6].in_features, n_classes)
  model.load_state_dict(torch.load(path_model, map_location=torch.device('cpu')))

    image = PIL.Image.open("U:\\PROJET_3A\\imagenette2-160\\train\\n01440764\\n01440764_17174.JPEG")
    transform = torchvision.transforms.Compose([torchvision.transforms.Resize(256),
                                                torchvision.transforms.CenterCrop(224),
                                                torchvision.transforms.ToTensor(),])
    image = transform(image)
    image = image[None, :]

    mean = [0.485, 0.456, 0.406]
    std = [0.229, 0.224, 0.225]

    normalize = torchvision.transforms.Normalize(mean=mean, std=std)
    grad_of_normalization = torchvision.transforms.Normalize(mean=0, std=std)
    target_class = 1

    test1 = image.clone().detach()
    test1.requires_grad = True
    vector_scores = self.model(normalize(test1))  
    loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
    loss_target.backward()
    grad_test1 = test1.grad.clone().detach()

    test2 = torch.autograd.Variable(normalize(image.clone().detach()), requires_grad=True)
    vector_scores = self.model(test2)  
    loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
    loss_target.backward()
    grad_test2 = test2.grad.clone().detach()
    grad_test2 = grad_of_normalization(grad_test2)


    test3 = normalize(image).clone().detach()
    test3.requires_grad = True
    vector_scores = self.model(test3)  
    loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
    loss_target.backward()
    grad_test3 = test3.grad.clone().detach()
    grad_test3 = grad_of_normalization(grad_test3)
2

Hi @AlexisAlexis,

Can you call self.model.zero_grad() in-between measurements? Gradients are accumulated in .backward(), so the result for your 2nd method is actually showing method 1 + method 2.

Also, can you share the results? How different are they?

3

Hi,
I added model.zero_grad() between the measurements and then displayed the gradients without modifying them with matplotlib. The differences are not huge but I see more intense spots in some measurements.

Example :

grads
grads1267×317 43.5 KB

4

Can you compare the error via its MSE or MAE value instead of a figure?

5

I added the calculation of the MSE, here are my results:

mse grad 1 grad 2 0.000966
mse grad 1 grad 3 0.000919
mse grad 2 grad 3 0.001061

Here is a part of my code after modification, don’t hesitate to tell me if something is wrong:

test1 = image.clone().detach()
test1.requires_grad = True
vector_scores = self.model(normalize(test1))  
loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
loss_target.backward()
grad_test1 = test1.grad.clone().detach()

self.model.zero_grad()

test2 = torch.autograd.Variable(normalize(image.clone().detach()), requires_grad=True)
vector_scores = self.model(test2)  
loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
loss_target.backward()
grad_test2 = test2.grad.clone().detach()
grad_test2 = grad_of_normalization(grad_test2)

self.model.zero_grad()

test3 = normalize(image.clone().detach())
test3.requires_grad = True
vector_scores = self.model(test3)  
loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
loss_target.backward()
grad_test3 = test3.grad.clone().detach()
grad_test3 = grad_of_normalization(grad_test3)

print("mse grad 1 grad 2 %f" % torch.nn.functional.mse_loss(grad_test1, grad_test2))
print("mse grad 1 grad 3 %f" % torch.nn.functional.mse_loss(grad_test1, grad_test3))
print("mse grad 2 grad 3 %f" % torch.nn.functional.mse_loss(grad_test2, grad_test3))
6

Solution : I forgot to set the model in eval mode with .eval()