Inception v3 RuntimeError with torch 1.0.0

[anianruoss]

I am running exactly the same code with torch==0.4.1 and torch==1.0.0.

torch==0.4.1: my code works with ResNet101 and Inception v3
torch==1.0.0: my code works with ResNet101 but Inception v3 fails with this stack trace:

Traceback (most recent call last):
  File "/home/anianruoss/experiments/imagenet/run_stadv_pytorch.py", line 100, in <module>
    flows_x0, optimizers[args.optimizer]
  File "/home/anianruoss/stAdv_pytorch/optimization.py", line 189, in pytorch_wrapper
    optim.step(closure)
  File "/home/anianruoss/venv/lib/python3.6/site-packages/torch/optim/adam.py", line 58, in step
    loss = closure()
  File "/home/anianruoss/stAdv_pytorch/optimization.py", line 183, in closure
    loss.backward()
  File "/home/anianruoss/venv/lib/python3.6/site-packages/torch/tensor.py", line 102, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/anianruoss/venv/lib/python3.6/site-packages/torch/autograd/__init__.py", line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

This is roughly what my code does:

optimizer = torch.optim.Adam([flows])

def closure():
    optimizer.zero_grad()
    loss = CustomLoss(flows).sum()
    loss.backward()
    return loss

for i in range(steps):
    optimizer.step(closure)

Do you know what could be the reason for this?

[vexilligera]

Did you solve it? I had the same issue I have no idea.

[Tony-Y]

Please try to use PyTorch 1.0.1.

[anianruoss]

Thank you for your help @Tony-Y. Unfortunately the problem remains even with torch==1.0.1.post2 and torchvision==0.2.1.

[Tony-Y]

Maybe, you have to modify CustomLoss slightly. An issue due to a custom loss has been reported:

One of the variables needed for gradient computation has been modified by an inplace operation_

I have made a custom loss function which is taking parameters calculated from a intermediate function ( a function which is taking as input the output of my CNN network). but from loss.backward() I am getting the error one of the variables needed for gradient computation has been modified by an inplace operation. I couldn’t find out the source of the error so I am embedding both functions. Loss function is class my_Loss(torch.nn.Module): def __init__(self): super(my_Loss,sel…

He resolved this problem by cloning tensors in the custom loss.

[anianruoss]

That seems to solve the problem. Thanks! However, I am still confused as to why this occurs with Inception v3 and not with ResNet-101.

[Tony-Y]

http://www.yongfengli.tk/2018/04/13/inplace-operation-in-pytorch.html

This article might help you to understand why it needs cloning.