# I got 'RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation' error **URL:** \ **Category:** Uncategorized\ **Created:** [February 3, 2019, 11:47am UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289 "2019-02-03T11:47:56Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![coincheung](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/32.png) [@coincheung](https://discuss.pytorch.org/u/coincheung)\ **Post date:** [February 3, 2019, 11:47am UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289/1 "2019-02-03T11:47:56Z")
I am trying online hard mining, and my simplified code is like this: ```python class OhemCELoss(nn.Module): def __init__ (self, thresh, n_min, ignore_lb=255, *args, **kwargs): super(OhemCELoss, self). __init__ () self.thresh = thresh self.n_min = n_min self.ignore_lb = ignore_lb self.criteria = nn.CrossEntropyLoss(ignore_index=ignore_lb) def forward(self, logits, labels): N, C, H, W = logits.size() n_pixs = N * H * W logits = logits.permute(0, 2, 3, 1).contiguous().view(-1, C) scores = F.softmax(logits, dim=1).cpu() labels = labels.view(-1) labels_cpu = labels.cpu() invalid_mask = labels_cpu==self.ignore_lb labels_cpu[invalid_mask] = 0 picks = scores[torch.arange(n_pixs), labels_cpu] picks[invalid_mask] = 1 sorteds, inds = torch.sort(picks) thresh = self.thresh if sorteds[self.n_min]thresh] = self.ignore_lb loss = self.criteria(logits, labels) return loss if __name__ == ' __main__': criteria1 = OhemCELoss(thresh=0.7, n_min=16*20*20//16).cuda() criteria2 = OhemCELoss(thresh=0.7, n_min=16*20*20//16).cuda() net1 = nn.Sequential( nn.Conv2d(3, 19, kernel_size=3, stride=2, padding=1), ) net1.cuda() net1.train() net2 = nn.Sequential( nn.Conv2d(3, 19, kernel_size=3, stride=2, padding=1), ) net2.cuda() net2.train() inten = torch.randn(16, 3, 20, 20).cuda() lbs = torch.randint(0, 19, [16, 20, 20]).cuda() lbs[1, 10, 10] = 255 logits1 = net1(inten) logits1 = F.interpolate(logits1, inten.size()[2:], mode='bilinear') logits2 = net2(inten) logits2 = F.interpolate(logits2, inten.size()[2:], mode='bilinear') loss1 = criteria1(logits1, lbs) loss2 = criteria2(logits2, lbs) loss = loss1 + loss2 loss.backward() ``` With this code I got the error message of: ```auto Traceback (most recent call last): File "loss.py", line 79, in loss.backward() File "/home/zhangzy/.local/lib/python3.5/site-packages/torch/tensor.py", line 102, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/home/zhangzy/.local/lib/python3.5/site-packages/torch/autograd/ __init__.py", line 90, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation ``` Where is the inplace operation that causes this error please? ---
**Author:** ![JuanFMontesinos](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/32/76115_2.png) [@JuanFMontesinos](https://discuss.pytorch.org/u/JuanFMontesinos)\ **Post date:** [February 3, 2019, 1:03pm UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289/2 "2019-02-03T13:03:46Z")
labels\_cpu[invalid\_mask] = 0 You arr masking them ---
**Author:** ![coincheung](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/32.png) [@coincheung](https://discuss.pytorch.org/u/coincheung)\ **Post date:** [February 3, 2019, 3:00pm UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289/3 "2019-02-03T15:00:16Z")
Would you please show me how to fix this? I changed it like this: ```python def forward(self, logits, labels): N, C, H, W = logits.size() n_pixs = N * H * W logits = logits.permute(0, 2, 3, 1).contiguous().view(-1, C) with torch.no_grad(): scores = F.softmax(logits, dim=1).cpu().detach() labels = labels.view(-1) labels_cpu = labels.cpu().detach() invalid_mask = labels_cpu==self.ignore_lb labels_cpu[invalid_mask] = 0 picks = scores[torch.arange(n_pixs), labels_cpu] picks[invalid_mask] = 1 sorteds, inds = torch.sort(picks) thresh = self.thresh if sorteds[self.n_min]thresh] = self.ignore_lb loss = self.criteria(logits, labels) return loss ``` But the problem still exists ---
**Author:** ![JuanFMontesinos](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/32/76115_2.png) [@JuanFMontesinos](https://discuss.pytorch.org/u/JuanFMontesinos)\ **Post date:** [February 3, 2019, 3:11pm UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289/4 "2019-02-03T15:11:34Z")
It’s not a matter of using that. I don’t know the content of those matrices but I expect you understand that masking is not backpropagable, thus, learnable. Problem there is that when you compute labels\_cpu you are moving labels to gpu but it still refers to labels in memory. You have to use Labels.cpu().clone().detach() So depending on what labels[picks\>thresh]= self.ignore does you may not be able to fix it. If you only want to compute loss over those features try to mask both, gt and labels, not to modify them ---
**Author:** ![coincheung](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/32.png) [@coincheung](https://discuss.pytorch.org/u/coincheung)\ **Post date:** [February 3, 2019, 3:20pm UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289/5 "2019-02-03T15:20:09Z")
I just need to change the labels according to the scores of the features. The features that are used to compute the loss are not modified when they are fed to the cross\_entropy. Since labels are input tensors, their values should be easily to be modified. I change the associated lines into: ```python scores = F.softmax(logits, dim=1).cpu().clone().detach() labels_cpu = labels.cpu().clone().detach() ``` But the problem still exists. How could I make it work please? ---
**Author:** ![JuanFMontesinos](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/32/76115_2.png) [@JuanFMontesinos](https://discuss.pytorch.org/u/JuanFMontesinos)\ **Post date:** [February 3, 2019, 3:23pm UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289/6 "2019-02-03T15:23:35Z")
What about this Line ? labels[picks\>thresh] = self.ignore\_lb You are assigning a value in place there, right? ---
**Author:** ![coincheung](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/32.png) [@coincheung](https://discuss.pytorch.org/u/coincheung)\ **Post date:** [February 3, 2019, 3:25pm UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289/7 "2019-02-03T15:25:16Z")
Yes, I have just found that without this line , the code can work. I just need to change some value of the labels into the ignored value. I tried to use `labels.requires_grad=False`, but it still cannot work. Why I cannot modify the labels please? ---
**Author:** ![coincheung](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/32.png) [@coincheung](https://discuss.pytorch.org/u/coincheung)\ **Post date:** [February 3, 2019, 3:29pm UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289/8 "2019-02-03T15:29:52Z")
I tried this: ```python labels[picks>thresh] = self.ignore_lb labels = labels.clone().detach() ``` and I works, I still cannot understand why the values of training labels cannot be changed . ---
**Author:** ![JuanFMontesinos](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/32/76115_2.png) [@JuanFMontesinos](https://discuss.pytorch.org/u/JuanFMontesinos)\ **Post date:** [February 3, 2019, 3:45pm UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289/9 "2019-02-03T15:45:26Z")
Because deep learning is based on computing gradients. If you just manually change the value of a tensor you cannot compute gradient because it simply does not exist. The closest thing you can do is Criteria(labels[p\>t],scores[p\>t]) You can compute the loss over those values, but you cannot modify the tensor unless you apply a continuous function over those values ---
**Author:** ![coincheung](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/32.png) [@coincheung](https://discuss.pytorch.org/u/coincheung)\ **Post date:** [February 3, 2019, 4:02pm UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289/10 "2019-02-03T16:02:14Z")
I see. Many thanks ! ---
**Author:** ![pgu-nd](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e56c9b/32.png) [@pgu-nd](https://discuss.pytorch.org/u/pgu-nd)\ **Post date:** [August 17, 2019, 5:22am UTC](https://discuss.pytorch.org/t/i-got-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-error/36289/11 "2019-08-17T05:22:16Z")
Could you please show me how to fix this problem? I changed labels = labels.clone() to labels = labels.clone().detach(), but it still not works. Thank you so much.