Runtime:Error:one of the variables needed for gradient computation has been modified by an inplace operation

[KFrank]

Hi Ashima!

@Ashima_Kalathingal

one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [100, 3]] is at version 2; expected version 1 instead.

The reported shape of the tensor, [100, 3], can be a useful hint – see
below.

For some explanation about how inplace-modification errors occur and
some techniques to debug them, see this post:

"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a

Hi Fahmyadan and Sangyoon! Here are some suggestions about how to track down (and maybe fix) inplace-modification errors. Note that an inplace modification in the forward pass is not necessarily* an error – it depends on whether and how the tensor that was modified is used in the backward pass. Note that inplace operations can be useful for saving memory – if you replace an innocent inplace operation with an out-of-place equivalent, your training will use more memory (and, to a minor e…

You can sometimes “automatically” fix such errors by using pytorch’s
sweep-inplace-modification-errors-under-the-rug context manager, but
it’s probably good practice to track down and understand the cause of
the error, even if you do use this solution.

      l = 100
      self.f = nn.Sequential(nn.Linear(3,l),nn.Tanh(),nn.Linear(l,1))

It’s quite possible that the shape-[100, 3] tensor reported in the error
message is self.f[0].weight (that is, Linear (3. l).weight). You
are probably training your NeuralODE and you should be aware that calling
optimizer.step() on your model will modify that Linear (3. l).weight
inplace. Are you using .backward (retain_graph = True) anywhere?

Best.

K. Frank