# Loss.backward throwing CUDA Errors

**URL:** <https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221>\
**Category:** autograd\
**Created:** [June 12, 2020, 2:32pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221 "2020-06-12T14:32:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![chinmay5](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/32/12682_2.png) [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\
**Post date:** [June 12, 2020, 2:32pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/1 "2020-06-12T14:32:05Z")

</div>

Sorry that I am asking this again but I would need a confirmation before I try to switch to the nightly builds for the issue since they seem to be a bit unstable.

This is the code:-

```auto
@staticmethod
    def backward(ctx, grad_output):
         grad_label = grad_output.clone()
        num_ft = grad_output.shape[0]
        # grad_label.data.resize_(num_ft, 32, 41)
        lin_indices_3d, lin_indices_2d = ctx.saved_variables
        num_ind = lin_indices_3d.data[0]
        grad_label.data.view(num_ft, -1).index_copy_(1, lin_indices_2d.data[1:1 + num_ind],
                                                     torch.index_select(grad_output.data.contiguous().view(num_ft, -1),
                                                                        1, lin_indices_3d.data[1:1 + num_ind]))
        # raw_input('sdflkj')
        return grad_label, None, None, None

```

The error I get is:-

 ![image](https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c098d9846d1edfef87f966410ff69db7f1c9e91.png)

The reference issue is:-

> [@CUDA Illegal Memory Access](https://discuss.pytorch.org/t/cuda-illegal-memory-access/85166):
>
> While trying to implement a backward pass, I keep getting the error of CUDA Illegal Memory Access. @staticmethod def backward(ctx, grad\_output): grad\_label = grad\_output.clone() num\_ft = grad\_output.shape[0] # grad\_label.data.resize\_(num\_ft, 32, 41) lin\_indices\_3d, lin\_indices\_2d = ctx.saved\_variables num\_ind = lin\_indices\_3d.data[0] grad\_label.data.view(num\_ft, -1).index\_copy\_(1, lin\_indices\_2d.data[1:1 + num\_ind], …

One of the solutions I saw was:-

> [@Autograd failed after add max\_pool1d layer](https://discuss.pytorch.org/t/autograd-failed-after-add-max-pool1d-layer/75812/6):
>
> Thanks for the code snippet. Since it’s not reproducible with the last code, could you try to use: x= F.max\_pool1d(x.transpose(1,2).contiguous(), kernel\_size=x.size()[1]) and rerun the script, please? Also, which PyTorch version are you using? Make sure to update to the latest stable release, as Variables are deprecated since 0.4.

However, even after ensuring the input is contiguous I got the same error. I am still not sure about the actual reason for the error though

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [June 12, 2020, 2:48pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/2 "2020-06-12T14:48:46Z")

</div>

Hi,

You can try to enable `torch.autograd.set_detect_anomaly(True)` to see which forward function corresponds to the failing one in the backward.

Also You can try to run the same code on CPU to see if you get a better error message.  
Also, I would double check all the indices in the multiple indexing you do.  
Finally. You should **never** use `.data` anymore. This will reduce the chance of weird bugs a lot 🙂

---

<div class="post-metadata">

**Author:** ![chinmay5](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/32/12682_2.png) [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\
**Post date:** [June 12, 2020, 2:55pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/3 "2020-06-12T14:55:31Z")

</div>

@albanD  
The `detetct_anamaly` takes me to this code snippet:-

 ![image](https://discuss.pytorch.org/uploads/default/original/3X/1/c/1ca712e98bead88ea68a40881db56c6ae0f87e31.png)

Which is related to MaxPool1D.

I think the indexing should be fine since this is a code snippet which worked on 0.4.0 version and I am trying to port it here.

Also, if I do not have to use **.data** what would be the alternative to ensure less bugs?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [June 12, 2020, 3:09pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/4 "2020-06-12T15:09:42Z")

</div>

Can you share the kernel size and image size that you give to the pooling layer?  
Can you also add just after it:

```auto
def hook(grad):
    print("Inside hook:")
    print(grad.size())
    print(grad.stride())
imageft.register_hook(hook)

```

To know exactly the gradient properties.  
Thanks!

For the `.data`, it depends what you use it for.  
If you want to get a new tensor detached from the previous one wrt to the autograd, use `.detach()`.  
If you want to do ops that are not tracked by the autograd inplace, use the context manager `with torch.no_grad()`.

---

<div class="post-metadata">

**Author:** ![chinmay5](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/32/12682_2.png) [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\
**Post date:** [June 12, 2020, 3:15pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/5 "2020-06-12T15:15:06Z")

</div>

@albanD  
Here are the values:-

Input shape = torch.Size([128, 442368, 5])

Also output for the code snippet you asked:-

```auto
Inside hook:
torch.Size([128, 442368, 1])
(442368, 1, 1)

```

Also for the **.data** as you can see here, I am taking the gradients, reshaping it and doing a mask select. I am sorry but I am not sure to which of the mentioned categories would this fall to?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [June 12, 2020, 3:40pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/6 "2020-06-12T15:40:57Z")

</div>

> Here are the values:-

And what is the kernel size?

> Also for the **.data** as you can see here

If you have no reason to use it, you can just remove it.  
It used to be useful when working with `Variable`s. But now that they don’t exist, you can just remove the `.data` 🙂

---

<div class="post-metadata">

**Author:** ![chinmay5](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/32/12682_2.png) [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\
**Post date:** [June 12, 2020, 3:42pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/7 "2020-06-12T15:42:23Z")

</div>

Kernel Size is 5. I basically remove the last dimension

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [June 12, 2020, 3:47pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/8 "2020-06-12T15:47:02Z")

</div>

Interesting.  
Running the following works fine for me, does it work for you as well?

```python
import torch

a = torch.rand([128, 442368, 5], device="cuda", requires_grad=True)

mod = torch.nn.MaxPool1d(5)

out = mod(a)

out.backward(torch.ones_like(out))

```

---

<div class="post-metadata">

**Author:** ![chinmay5](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/32/12682_2.png) [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\
**Post date:** [June 12, 2020, 3:55pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/9 "2020-06-12T15:55:42Z")

</div>

@albanD  
Here is what I get when trying the same thing:-

 ![image](https://discuss.pytorch.org/uploads/default/original/3X/5/5/55e7b3820c5aaf37d86b99bf85e225bee34d583d.png)

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [June 12, 2020, 4:02pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/10 "2020-06-12T16:02:05Z")

</div>

Ah ! Making progress 🙂

So with the `Quadro GP100` and a source install I can’t reproduce that.

Do you see the same behavior on nightly build for this test script?  
Which version of cuda are you using and how did you installed pytorch?

---

<div class="post-metadata">

**Author:** ![chinmay5](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/32/12682_2.png) [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\
**Post date:** [June 12, 2020, 4:05pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/11 "2020-06-12T16:05:42Z")

</div>

@albanD  
I have not switched to nightly builds yet. I am using PyTorch 1.2 although I tried updating uptil 1.5 and error persists. In all the cases, I ended up using pip instllations.

As for the OS, it is Ubuntu 18.04 and the Cuda version is  
 ![image](https://discuss.pytorch.org/uploads/default/original/3X/3/2/32de47cee50185a6670d1ae32febf16b854d6626.png)

![image](https://discuss.pytorch.org/uploads/default/original/3X/e/7/e70c7e5425ab3c5a3d107f41fc79b3e71e768020.png)

![image](https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef986367d3702a78225c847a543c7b00d9414989.png)

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [June 12, 2020, 4:09pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/12 "2020-06-12T16:09:12Z")

</div>

Thanks for all the info.

Can you try to create a temporary python virtual env, install nightly and try to run the simple repro script. above?

---

<div class="post-metadata">

**Author:** ![chinmay5](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/32/12682_2.png) [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\
**Post date:** [June 12, 2020, 4:41pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/13 "2020-06-12T16:41:38Z")

</div>

@albanD I created a new environment.

 ![image](https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf9505b7c3435d1df380c93ec50b3333726981cc.png)

In the given case, I still get an error

 ![image](https://discuss.pytorch.org/uploads/default/original/3X/d/7/d7fe2de4acd6a8fce936f0d5b14b640c8b551b64.png)

---

<div class="post-metadata">

**Author:** ![chinmay5](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/32/12682_2.png) [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\
**Post date:** [June 12, 2020, 5:16pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/14 "2020-06-12T17:16:28Z")

</div>

I attached the results. Do you think I am making some mistake?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [June 12, 2020, 5:17pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/15 "2020-06-12T17:17:25Z")

</div>

@ptrblck do you have an RTX 2070 somewhere to try and repro this by any chance? 😃

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [June 12, 2020, 5:19pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/16 "2020-06-12T17:19:00Z")

</div>

I just found [https://github.com/pytorch/pytorch/issues/38764](https://github.com/pytorch/pytorch/issues/38764) which seem to claim that this was fixed.  
@ptrblck will know better!

---

<div class="post-metadata">

**Author:** ![chinmay5](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/32/12682_2.png) [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\
**Post date:** [June 12, 2020, 5:44pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/17 "2020-06-12T17:44:33Z")

</div>

I made a mistake. I was able to run the code snippet without error on the nightly build. I will try to see if it works on the entire code next and will update you asap

---

<div class="post-metadata">

**Author:** ![chinmay5](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/32/12682_2.png) [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\
**Post date:** [June 12, 2020, 6:14pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/18 "2020-06-12T18:14:48Z")

</div>

@albanD it works with the nightly version. Thank you so so much for all your support. I am pretty sure I could not have figured it out without your help

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [June 12, 2020, 7:27pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/19 "2020-06-12T19:27:03Z")

</div>

Sorry for the delayed reply, but yes it should be fixed by now. Let us know, if you run into this issue again, please.

---

<div class="post-metadata">

**Author:** ![chinmay5](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay5/32/12682_2.png) [@chinmay5](https://discuss.pytorch.org/u/chinmay5)\
**Post date:** [June 12, 2020, 7:28pm UTC](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221/20 "2020-06-12T19:28:10Z")

</div>

Just a query, when would a stable version be out. I am not sure about the stability of the nightly builds ☹

[Next page](https://discuss.pytorch.org/t/loss-backward-throwing-cuda-errors/85221.md?page=2)