# Where is the nn.Linear cuda implementation

**URL:** <https://discuss.pytorch.org/t/where-is-the-nn-linear-cuda-implementation/128310>\
**Category:** C++\
**Created:** [August 2, 2021, 9:05am UTC](https://discuss.pytorch.org/t/where-is-the-nn-linear-cuda-implementation/128310 "2021-08-02T09:05:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![cheng\_chen2](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cheng_chen2/32/31715_2.png) [@cheng\_chen2](https://discuss.pytorch.org/u/cheng_chen2)\
**Post date:** [August 2, 2021, 9:05am UTC](https://discuss.pytorch.org/t/where-is-the-nn-linear-cuda-implementation/128310/1 "2021-08-02T09:05:16Z")

</div>

I cannot find the cuda implementation of nn.Linear, for forward implementation and backward implementation. Can anyone helps me with this?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [August 8, 2021, 6:46am UTC](https://discuss.pytorch.org/t/where-is-the-nn-linear-cuda-implementation/128310/2 "2021-08-08T06:46:30Z")

</div>

`nn.Linear` is using `F.linear`, which will call into `addmm` or `matmul` [here](https://github.com/pytorch/pytorch/blob/4a15f4a902c5640f3be4d18027db4316dc11d6d9/aten/src/ATen/native/Linear.cpp#L33-L40) and then deploy to the cublas method [here](https://github.com/pytorch/pytorch/blob/c1c9be16c4d0648fc134d04f30c8463575df7ada/aten/src/ATen/native/cuda/Blas.cpp).

---

<div class="post-metadata">

**Author:** ![cheng\_chen2](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cheng_chen2/32/31715_2.png) [@cheng\_chen2](https://discuss.pytorch.org/u/cheng_chen2)\
**Post date:** [August 18, 2021, 7:05am UTC](https://discuss.pytorch.org/t/where-is-the-nn-linear-cuda-implementation/128310/3 "2021-08-18T07:05:21Z")

</div>

Thanks for the answer. And btw, what is the backward function/implementation for the F.linear? I see many Ops defined their related backward functions in native\_functions.yaml file, but for thoes not defined , how do we check their backward implmenetation? Is there a common way?

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [August 18, 2021, 7:17am UTC](https://discuss.pytorch.org/t/where-is-the-nn-linear-cuda-implementation/128310/4 "2021-08-18T07:17:50Z")

</div>

The backward methods of the aforementioned functions are defined in the `derivatives.yaml` as seen [here](https://github.com/pytorch/pytorch/blob/d431c77d76b8c92dfddfdfec2ce29b2e46c441c2/tools/autograd/derivatives.yaml#L204-L226).

---

<div class="post-metadata">

**Author:** ![Ziyu\_Huang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ziyu_huang/32/14158_2.png) [@Ziyu\_Huang](https://discuss.pytorch.org/u/Ziyu_Huang)\
**Post date:** [June 10, 2022, 8:16am UTC](https://discuss.pytorch.org/t/where-is-the-nn-linear-cuda-implementation/128310/5 "2022-06-10T08:16:44Z")

</div>

Thank you very much for your answer!!! But I still have some small questions:  
in [https://github.com/pytorch/pytorch/blob/c1c9be16c4d0648fc134d04f30c8463575df7ada/aten/src/ATen/native/cuda/Blas.cpp](https://github.com/pytorch/pytorch/blob/c1c9be16c4d0648fc134d04f30c8463575df7ada/aten/src/ATen/native/cuda/Blas.cpp) I can not find **function named addmm, or matmul** , only can find something like, addmm\_out\_cuda\_impl. I know they should be, close, but how they are related together?

Given that I am implementing linear layer myself, I also wonder, **how can I use** this in my code, which is, can I really include and run them?

Thank you!!!

---

<div class="post-metadata">

**Author:** ![Karthik\_Ganesan](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/karthik_ganesan/32/34393_2.png) [@Karthik\_Ganesan](https://discuss.pytorch.org/u/Karthik_Ganesan)\
**Post date:** [June 10, 2022, 4:58pm UTC](https://discuss.pytorch.org/t/where-is-the-nn-linear-cuda-implementation/128310/6 "2022-06-10T16:58:02Z")

</div>

Hi @Ziyu_Huang , I had the same issue so I wrote code to implement the Conv2D and Linear layers using native CUDA code [here](https://github.com/KarthikGanesan88/pytorch-manual-layers). Hopefully it will be useful for you! I only wrote code for the forward pass in CUDA but hopefully you can use that as a base to write code for backward if you need that.