RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [100, 2, 6]], which is output 0 of SigmoidBackward0, is at version 6; expected version 0 instead. Hint: the backtrace fur

[KFrank]

Hi Talon!

@talonx1

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [100, 2, 6]], which is output 0 of SigmoidBackward0, is at version 6; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

The error message gives you information that can point you in the right direction.

The tensor that is being modified inplace has shape [100, 2, 6] and appears
to be being passed into a Sigmoid. See if you can find that tensor in your code.

The error message complains about “version 6,” so the tensor is being modified
inplace six times before you call .backward(). That may give you a hint about
what operation is performing the inplace modification. You can also print out
the ._version property of the suspect tensor at various places in your code
and use a divide-and-conquer scheme to track down where exactly the tensor
is being modified.

Here’s a post that discusses some debugging techniques for these kinds of
errors:

"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a

Hi Fahmyadan and Sangyoon! Here are some suggestions about how to track down (and maybe fix) inplace-modification errors. Note that an inplace modification in the forward pass is not necessarily* an error – it depends on whether and how the tensor that was modified is used in the backward pass. Note that inplace operations can be useful for saving memory – if you replace an innocent inplace operation with an out-of-place equivalent, your training will use more memory (and, to a minor e…

Good luck!

K. Frank