One of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [32, 2]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead

[KFrank]

Hi Varun!

@varun_work

one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [32, 2]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead.

This error message is telling you (from the tensor shape) that
block3[3].weight is being modified inplace.

    self.optimizer3.step()

optimizer3.step() modifies inplace the Parameters it is optimizing.

    loss = self.alpha * l1 + (1 - self.alpha) * l2
    ...
    loss.backward()

What’s going on is that you modify block3. But loss depends on l2,
so when you call loss.backward(), you backpropagate through block3
again, hence the error.

(You have other similar errors, but when this first error is detected, the
call to .backward() exits.)

I don’t understand the rationale behind what you are doing, but a fix might
be as simple as first calling all of your .zero_grad()s and .backward()s
and then calling all of your optimizer.step()s.

Please also take a look at this post that explains how to debug such
inplace-modification errors:

"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a

Hi Fahmyadan and Sangyoon! Here are some suggestions about how to track down (and maybe fix) inplace-modification errors. Note that an inplace modification in the forward pass is not necessarily* an error – it depends on whether and how the tensor that was modified is used in the backward pass. Note that inplace operations can be useful for saving memory – if you replace an innocent inplace operation with an out-of-place equivalent, your training will use more memory (and, to a minor e…

Best.

K. Frank