RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [19, 140, 32]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace furt

[KFrank]

Hi Herbert!

@Dragon_Herbert

  File "/trajectron_workspace/STAR/src/star.py", line 462, in forward
    self.relu(self.input_embedding_layer_temporal(nodes_current)))
...
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [19, 140, 32]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead.

I don’t see the cause of your inplace-modification error, but here are some
things to look at:

Can you locate a tensor of shape [19, 140, 32], perhaps the output of
the self.relu() in line 462?

Try checking its ._version property right after the relu() and then again
right before you call loss.backward(). If its ._version changes, then it’s
being modified inplace. If ._version changes from 0 to 1, then you most
likely have found your culprit, as that agrees with the version mismatch that
autograd is reporting.

For some discussion about what can cause inplace-modification errors
and how to find and fix them, see this post:

"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a

Hi Fahmyadan and Sangyoon! Here are some suggestions about how to track down (and maybe fix) inplace-modification errors. Note that an inplace modification in the forward pass is not necessarily* an error – it depends on whether and how the tensor that was modified is used in the backward pass. Note that inplace operations can be useful for saving memory – if you replace an innocent inplace operation with an out-of-place equivalent, your training will use more memory (and, to a minor e…

Best.

K. Frank