Hi.

I’m trying to make adversarial examples with iterative method but found out that resultant adversarial examples are very different from each other although they were all made under the same condition. Does anybody know what is the problem? Is it precision issue?

The following codes are just templates to show the mentioned problem, not the exact code

``````import torch
import torch.nn.functional as F
import torchvision.models as models

mean = torch.cuda.FloatTensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std = torch.cuda.FloatTensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
def normalizer(tensor):
return (tensor-mean)/std

res101 = models.resnet101(pretrained=True)
res101.eval()
res101.cuda();

base = torch.zeros(1,3,224,224).cuda()
input1 = base.clone()
input2 = base.clone()
K = 200

for k in range(K):
normed_input1 = normalizer(input1)
loss1 = res101(normed_input1).sum()

loss1.backward()

normed_input2 = normalizer(input2)
loss2 = res101(normed_input2).sum()

loss2.backward()

if k% 20 == 0:
print('step :', k+1, 'close :', torch.allclose(input1, input2), 'cosine similarity :', F.cosine_similarity(input1.view(-1), input2.view(-1), dim=0).item())
``````

and here are the results

``````step : 1 close : True cosine similarity : 1.0
step : 21 close : False cosine similarity : 0.9865298271179199
step : 41 close : False cosine similarity : 0.9560290575027466
step : 61 close : False cosine similarity : 0.8762256503105164
step : 81 close : False cosine similarity : 0.7901834845542908
step : 101 close : False cosine similarity : 0.711552619934082
step : 121 close : False cosine similarity : 0.6323087811470032
step : 141 close : False cosine similarity : 0.5666524171829224
step : 161 close : False cosine similarity : 0.4950137734413147
step : 181 close : False cosine similarity : 0.44248107075691223
``````

What happens when you zero_grad before the forward? I.e.,
loss1= res101(normed_input).sum()

I tried as you explained, but it makes no difference.

Also, res101 should be set to training mode right? I.e., res101.training().

Otherwise, if you set input1 = Variable(torch.zeros(1,3,224,224),requires_grad=True) and same for input2 on every iteration then it should work…

@Ranahanocka

Can you explain more about why res101 should be set to training mode?

As you see, I add `input1 = Variable(input1.data, requires_grad=True)` in for loop.
Also I set `res101.eval()` because I don’t want to enable some stochastic behaviors that are encouraged during model training, e.g. dropout and batchnorm. and also we can still get gradients from the model when it is in eval mode.

You need to have the model parameter gradients for backprop (i.e., the chain rule) with the autograd.Variable.

See more here:

We still can get gradients from model parameters when we set `resnet101.eval()` because eval mode is to deactivate above-mentioned stochastic modules like dropout or batchnorm. it does not deactivate autograd engine.