RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 6890]], which is output 0 of SelectBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detect

[KFrank]

Hi July!

@julyaugust12345

inplace operation: [torch.cuda.FloatTensor [1, 6890]]

Can you identify a tensor in your forward pass that has shape [1, 6890]?
See if you can find where it is being modified.

Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Try using set_detect_anomaly (True). Doing so provides additional
information that can be very helpful in tracking down the cause of your
error.

    def get_pose(self, text_feature: Tensor) -> Tensor:

There’s a lot of code that you haven’t posted that could be causing your
error.

    def get_pose(self, text_feature: Tensor) -> Tensor:
        ...
        return pose_padding(pose.data).to(self.device)

You’ve posted two nearly identical versions of get_pose() (and we can’t
tell which one is being run when you get your error).

Note, this second version uses pose.data. .data is deprecated (in the
public api) and can lead to errors, so you shouldn’t be using it.

For suggestions about how to find and fix inplace-modification errors,
please see the following post:

"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a

Hi Fahmyadan and Sangyoon! Here are some suggestions about how to track down (and maybe fix) inplace-modification errors. Note that an inplace modification in the forward pass is not necessarily* an error – it depends on whether and how the tensor that was modified is used in the backward pass. Note that inplace operations can be useful for saving memory – if you replace an innocent inplace operation with an out-of-place equivalent, your training will use more memory (and, to a minor e…

Good luck!

K. Frank