# Defining Autograd functions tutorial - typo? **URL:** \ **Category:** Uncategorized\ **Created:** [June 24, 2020, 11:41pm UTC](https://discuss.pytorch.org/t/defining-autograd-functions-tutorial-typo/86797 "2020-06-24T23:41:39Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Sam\_Lerman](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/32/37165_2.png) [@Sam\_Lerman](https://discuss.pytorch.org/u/Sam_Lerman)\ **Post date:** [June 24, 2020, 11:41pm UTC](https://discuss.pytorch.org/t/defining-autograd-functions-tutorial-typo/86797/1 "2020-06-24T23:41:39Z")
Hi, I just read through this clear and straightforward tutorial on defining custom Autograd functions in Pytorch: [https://pytorch.org/tutorials/beginner/examples\_autograd/two\_layer\_net\_custom\_function.html](https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html) The code is very clear, but one thing I’d like to clarify on is the comment in the code that says “In the backward pass we receive a Tensor containing the gradient of the loss with respect to the output, and we need to compute the gradient of the loss with respect to the input.” Shouldn’t it be “we need to compute the gradient of the _output_ with respect to the input”? I’m confused how we could define the gradient of the loss with respect to the input, since typically we think of the gradient of a function as the gradient of that function’s output with respect to its input. Forgive me if I misunderstood, but I’m also somewhat confused about what exactly is referred to by “gradient of the loss with respect to the output”. In the formulation of backprop that I’m familiar with, the function’s gradient is simply multiplied to the recursive sum of forward gradients (to put it into very loose English). I could give a very short and hopefully simple formal summary of that formulation if needed; it would definitely help me to understand how Pytorch’s autograd fits into my existing understanding. Thanks so much for any further clarifications! ---
**Author:** ![harsha\_g](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/4491bb/32.png) [@harsha\_g](https://discuss.pytorch.org/u/harsha_g)\ **Post date:** [June 24, 2020, 11:49pm UTC](https://discuss.pytorch.org/t/defining-autograd-functions-tutorial-typo/86797/2 "2020-06-24T23:49:31Z")
Your loss `L` is a function of `y` (target) and `y_hat`(output). The first step in backpropagation is `dL/dy_hat` (gradient of the loss with respect to the output) and that’s what you are reading in the tutorial. ---
**Author:** ![Sam\_Lerman](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/32/37165_2.png) [@Sam\_Lerman](https://discuss.pytorch.org/u/Sam_Lerman)\ **Post date:** [June 24, 2020, 11:51pm UTC](https://discuss.pytorch.org/t/defining-autograd-functions-tutorial-typo/86797/3 "2020-06-24T23:51:03Z")
In the general case, for a function not belonging to the output layer, would the correct interpretation be “we need to compute the gradient of the _output_ with respect to the input”? ---
**Author:** ![harsha\_g](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/4491bb/32.png) [@harsha\_g](https://discuss.pytorch.org/u/harsha_g)\ **Post date:** [June 24, 2020, 11:53pm UTC](https://discuss.pytorch.org/t/defining-autograd-functions-tutorial-typo/86797/4 "2020-06-24T23:53:13Z")
Sure, that’s a reasonable interpretation. ---
**Author:** ![Sam\_Lerman](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/32/37165_2.png) [@Sam\_Lerman](https://discuss.pytorch.org/u/Sam_Lerman)\ **Post date:** [June 24, 2020, 11:54pm UTC](https://discuss.pytorch.org/t/defining-autograd-functions-tutorial-typo/86797/5 "2020-06-24T23:54:12Z")
Let me clarify. The ReLU function is `f(x)=max(0,x)` . It means if `x<=0` then `f(x)=0` , else `f(x)=x` . In the first case, when `x<0` , the derivative of `f(x)` with respect to `x` is `f'(x)=0` . So, we perform `grad_input[input < 0] = 0` . In other words, we "compute the gradient of the _output_ with respect to the input” and not "compute the gradient of the loss with respect to the input”. ---
**Author:** ![Sam\_Lerman](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/32/37165_2.png) [@Sam\_Lerman](https://discuss.pytorch.org/u/Sam_Lerman)\ **Post date:** [June 24, 2020, 11:54pm UTC](https://discuss.pytorch.org/t/defining-autograd-functions-tutorial-typo/86797/6 "2020-06-24T23:54:44Z")
Oh okay, sorry, didn’t see your reply before posting. So my understanding in that regard is correct? ---
**Author:** ![harsha\_g](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/4491bb/32.png) [@harsha\_g](https://discuss.pytorch.org/u/harsha_g)\ **Post date:** [June 24, 2020, 11:55pm UTC](https://discuss.pytorch.org/t/defining-autograd-functions-tutorial-typo/86797/7 "2020-06-24T23:55:25Z")
Not in the original post but yes in the next post 😉 . ---
**Author:** ![Sam\_Lerman](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/32/37165_2.png) [@Sam\_Lerman](https://discuss.pytorch.org/u/Sam_Lerman)\ **Post date:** [June 24, 2020, 11:59pm UTC](https://discuss.pytorch.org/t/defining-autograd-functions-tutorial-typo/86797/8 "2020-06-24T23:59:11Z")
Not sure which posts you’re referring to, I don’t think my understanding has changed. But I’m going to assume we’re on the same page about the meaning of the second half of the comment 🙂 as for the first half of the comment, “gradient of the loss with respect to the output”, is there a general-case interpretation for that as well? I need to use a custom Autograd function that belongs to arbitrary hidden layers, not just output layers. Would it help if I gave some quick backprop formulas so that we could synchronize our language in a formal way? ---
**Author:** ![Sam\_Lerman](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_lerman/32/37165_2.png) [@Sam\_Lerman](https://discuss.pytorch.org/u/Sam_Lerman)\ **Post date:** [June 25, 2020, 12:22am UTC](https://discuss.pytorch.org/t/defining-autograd-functions-tutorial-typo/86797/9 "2020-06-25T00:22:14Z")
Basically, I’m trying to clarify how the variable `grad_output` in that tutorial is computed. Is there an explicit formula written anywhere? If not, I can quickly derive backprop in simple terms and maybe we can identify where `grad_output` fits into that derivation. It really wouldn’t be as complicated as it sounds. I just need to figure out whether I can use this for my research. ---
**Author:** ![harsha\_g](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/4491bb/32.png) [@harsha\_g](https://discuss.pytorch.org/u/harsha_g)\ **Post date:** [June 25, 2020, 5:36am UTC](https://discuss.pytorch.org/t/defining-autograd-functions-tutorial-typo/86797/10 "2020-06-25T05:36:40Z")
@Sam_Lerman I can assure you my knowledge of Autograd is atrocious 😁. However, just in case, did you take a look at this post 👉 [Exact meaning of grad\_input and grad\_output](https://discuss.pytorch.org/t/exact-meaning-of-grad-input-and-grad-output/14186/9)? On a side note, the two posts I was referring to were on either side of my first reply. Your original post and your subsequent post to my reply. In my minimal understanding of backpropagation and automatic differentiation, I would not necessarily think the docstring you were referring to is either wrong or misleading. It’s best left to the individual to interpret it in a way that could aid them in generalizing their intuition (which could be wrong too 😉 ).