RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [200, 2]], which is output 0 of AsStridedBackward0, is at version 1401; expected version 1400 instead

[KFrank]

Hi Lukas!

@Lukas_Scholz

When trying to to a backward-pass in my model I receive the following error:

First, try using the inplace-modification-error debugging suggestions in the
following post:

"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a

Hi Fahmyadan and Sangyoon! Here are some suggestions about how to track down (and maybe fix) inplace-modification errors. Note that an inplace modification in the forward pass is not necessarily* an error – it depends on whether and how the tensor that was modified is used in the backward pass. Note that inplace operations can be useful for saving memory – if you replace an innocent inplace operation with an out-of-place equivalent, your training will use more memory (and, to a minor e…

clue.py

class CLUE(BaseNet):
    ...
    def optimise(self, min_steps=3, max_steps=25,
                 n_early_stop=3):
            objective.sum(dim=0).backward()

            self.optimizer.step()
            ...

Note that optimizer.step() counts as an inplace modification for the parameters
being optimized. Parts of your post have the flavor of optimizer.step() being
the cause of the inplace modification.

gaussian_mlp.py

class GaussianMLP(nn.Module):
    def __init__(self, input_dim, width, depth, output_dim, flatten_image):
        ...
        layers = [nn.Linear(input_dim, width), nn.ReLU()]
        for i in range(depth - 1):
            layers.append(nn.Linear(width, width))
            layers.append(nn.ReLU())
        layers.append(nn.Linear(width, 2 * output_dim))
        ...

Full Traceback of the Error Message:

...
  File "/Users/lukasscholz/repositorys/counterfactuals/counterfactual_xai/utils/clue/gaussian_mlp.py", line 27, in forward
    x = self.block(x)
  ...
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 116, in forward
    return F.linear(input, self.weight, self.bias)
 (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:118.)
  ...
    objective.sum(dim=0).backward()  # backpropagate
  ...
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [200, 2]], which is output 0 of AsStridedBackward0, is at version 1401; expected version 1400 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

This error message is telling you that the tensor that was modified inplace has
shape [200, 2], and it appears to be a Linear. Does GaussianMLP.block
contain a Linear (200, 2), perhaps the last Linear in the Sequential?

(Note, autograd raises the error on the first inplace modification it detects, so,
if this is what’s going on, earlier Linears in block probably also have been
modified inplace.)

Anyway, given the error message you posted, start by locating any tensors of
shape [200, 2] you have (call them generically t) and print out t._version
at various places in your code to use a divide-and-conquer strategy (binary
search) to locate where t._version changes from 1400 to 1401. This is
where the relevant inplace modification occurs.

If optimizer.step() is the cause, then ask yourself if you are ever calling
.backward() on a quantity that depends on some “old” data, that is, that
depends (in the sense of a still-live autograd computation graph) on some
data that was computed using some model Parameter (perhaps that Linear)
prior to the most recent optimizer.step() call.

(As an aside, .data has been deprecated for public-facing use. Using it can
confuse autograd and cause errors. I didn’t notice any .data calls in your
code that looked like they would be causing trouble, but you should rewrite
your code to eliminate them anyway, if only because the semantics of .data
could silently change in future versions.)

Good luck!

K. Frank