# Defining Custom leaky\_relu functions **URL:** \ **Category:** autograd\ **Created:** [March 9, 2020, 11:53am UTC](https://discuss.pytorch.org/t/defining-custom-leaky-relu-functions/72576 "2020-03-09T11:53:24Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![111179](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111179/32/22195_2.png) [@111179](https://discuss.pytorch.org/u/111179)\ **Post date:** [March 9, 2020, 11:53am UTC](https://discuss.pytorch.org/t/defining-custom-leaky-relu-functions/72576/1 "2020-03-09T11:53:24Z")
I try to defining custom leaky\_relu function base on autograd, but the code shows “function MyReLUBackward returned an incorrect number of gradients (expected 2, got 1)”, can you give me some advice? Thank you so much for your help. the code as shown: import torch from torch.autograd import Variable import math class MyReLU(torch.autograd.Function): @staticmethod def forward(ctx, input,negative\_slope): output = input.clamp(min=0)+input.clamp(max=0)\*negative\_slope ctx.save\_for\_backward(input) return output @staticmethod def backward(ctx, grad\_output): input, = ctx.saved\_tensors negative\_slope, = ctx.saved\_tensors grad\_input = grad\_output.clone() return grad\_input dtype = torch.float N, D\_in, H, D\_out = 64, 1000, 100, 10 x = torch.randn(N, D\_in, device=device, dtype=dtype) y = torch.randn(N, D\_out, device=device, dtype=dtype) w1 = torch.randn(D\_in, H, device=device, dtype=dtype, requires\_grad=True) w2 = torch.randn(H, D\_out, device=device, dtype=dtype, requires\_grad=True) learning\_rate = 1e-6 for t in range(500): relu = MyReLU.apply y\_pred = relu(x.mm(w1),0.01).mm(w2) loss = (y\_pred - y).pow(2).sum() if t % 100 == 99: print(t, loss.item()) loss.backward() with torch.no\_grad(): w1 -= learning\_rate \* w1.grad w2 -= learning\_rate \* w2.grad w1.grad.zero\_() w2.grad.zero\_() ---
**Author:** ![DAlolicorn](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/ecb155/32.png) [@DAlolicorn](https://discuss.pytorch.org/u/DAlolicorn)\ **Post date:** [March 9, 2020, 1:25pm UTC](https://discuss.pytorch.org/t/defining-custom-leaky-relu-functions/72576/2 "2020-03-09T13:25:17Z")
Hi, the problem is that the backward function expect you to return all gradients for each input argument of forward function. Since you don’t need gradient for the argument negative\_slope, just return None: ```auto @staticmethod def forward(ctx, input,negative_slope): output = input.clamp(min=0)+input.clamp(max=0)*negative_slope ctx.save_for_backward(input) ctx.slope = negative_slope return output @staticmethod def backward(ctx, grad_output): input, = ctx.saved_tensors slope = ctx.slope grad_input = grad_output.clone() grad_input = grad_input * (input > 0).float() + grad_input * (input < 0).float() * slope return grad_input, None ``` By the way corrected the backward and forward function without testing, hope it works. ---
**Author:** ![111179](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111179/32/22195_2.png) [@111179](https://discuss.pytorch.org/u/111179)\ **Post date:** [March 10, 2020, 12:59am UTC](https://discuss.pytorch.org/t/defining-custom-leaky-relu-functions/72576/3 "2020-03-10T00:59:54Z")
yep! Thank you !!! 😁 ---
**Author:** ![Chinmay\_Rane](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay_rane/32/35162_2.png) [@Chinmay\_Rane](https://discuss.pytorch.org/u/Chinmay_Rane)\ **Post date:** [February 27, 2021, 4:44pm UTC](https://discuss.pytorch.org/t/defining-custom-leaky-relu-functions/72576/4 "2021-02-27T16:44:32Z")
Hi thank you the custom leaky. I was wondering. if i trained the negative slope value here, where please correct me if i am wrong. I need to pass the gradient required for the slope in backward propagation as i did below after calculating the gradient for slope @staticmethod def forward(ctx, input,negative\_slope): output = input.clamp(min=0)+input.clamp(max=0)\*negative\_slope ctx.save\_for\_backward(input) ctx.slope = negative\_slope return output @staticmethod def backward(ctx, grad\_output): input, = ctx.saved\_tensors slope = ctx.slope grad\_input = grad\_output.clone() grad\_input = grad\_input \* (input \> 0).float() + grad\_input \* (input \< 0).float() \* slope ``` return grad_input, negative_slope_gradient ``` how can i get the updated negative slope after training and also, how can i check if it is training simultaneously. I, also understand we need to do negative\_slope.requires\_grad = True. Any help is appreciated. Thank you ---
**Author:** ![111179](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111179/32/22195_2.png) [@111179](https://discuss.pytorch.org/u/111179)\ **Post date:** [March 3, 2021, 5:29am UTC](https://discuss.pytorch.org/t/defining-custom-leaky-relu-functions/72576/5 "2021-03-03T05:29:22Z")
Hi, the `negative_slope` is a constant as input not learnable parameters. If you want to implement a ReLU function with learnable parameters, you can refer to PReLU, for example: [https://medium.com/@shoray.goel/prelu-activation-e294bb21fefa](https://medium.com/@shoray.goel/prelu-activation-e294bb21fefa) Hope it can help you! ---
**Author:** ![Chinmay\_Rane](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay_rane/32/35162_2.png) [@Chinmay\_Rane](https://discuss.pytorch.org/u/Chinmay_Rane)\ **Post date:** [March 9, 2021, 4:24pm UTC](https://discuss.pytorch.org/t/defining-custom-leaky-relu-functions/72576/6 "2021-03-09T16:24:11Z")
Hi thanks for your response. the medium post was helpful. I designed a trainable parameter. . I have question on the above leaky relu. have you tried running it on cuda? On CPU the forward and backward both work but when you shift device to cuda the code doesn’t access the backward function, it only access the forward pass. I have also created a new question in the hopes to get the answer here [Custom Backward function using Function from torch.autograd fails on cuda but works on cpu](https://discuss.pytorch.org/t/custom-backward-function-using-function-from-torch-autograd-fails-on-cuda-but-works-on-cpu/114157) Please let me know if it works for you. It definitely doesn’t work for me. I even tried the make\_dirty → ctx.mark\_dirty(output). I get the following error -\>RuntimeError: a leaf Variable that requires grad has been used in an in-place operation. then i also tried the ‘output.view\_as(output)’ but i got ‘RuntimeError: Some elements marked as dirty during the forward method were not returned as output. The inputs that are modified inplace must all be outputs of the Function.’ ---
**Author:** ![111179](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111179/32/22195_2.png) [@111179](https://discuss.pytorch.org/u/111179)\ **Post date:** [March 12, 2021, 6:28am UTC](https://discuss.pytorch.org/t/defining-custom-leaky-relu-functions/72576/7 "2021-03-12T06:28:36Z")
Sorry, I haven’t try it on GPU. But, I implement another custom activation function on GPU, for instance: class Surrogate\_BP\_Function(torch.autograd.Function): ```auto @staticmethod def forward(ctx, input): ctx.save_for_backward(input) out = torch.zeros_like(input).cuda() out[input > 0] = 1.0 return out @staticmethod def backward(ctx, grad_output): input, = ctx.saved_tensors grad_input = grad_output.clone() grad = grad_input * 0.3 * F.threshold(1.0 - torch.abs(input), 0, 0) return grad ``` It works on GPU. I hope it can help you. ---
**Author:** ![Chinmay\_Rane](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chinmay_rane/32/35162_2.png) [@Chinmay\_Rane](https://discuss.pytorch.org/u/Chinmay_Rane)\ **Post date:** [March 13, 2021, 2:49am UTC](https://discuss.pytorch.org/t/defining-custom-leaky-relu-functions/72576/8 "2021-03-13T02:49:38Z")
> [@111179](#): > > ```auto > @staticmethod > def forward(ctx, input): > ctx.save_for_backward(input) > out = torch.zeros_like(input).cuda() > out[input > 0] = 1.0 > return out > > @staticmethod > def backward(ctx, grad_output): > input, = ctx.saved_tensors > grad_input = grad_output.clone() > grad = grad_input * 0.3 * F.threshold(1.0 - torch.abs(input), 0, 0) > return grad > > ``` hey thanks for helping but i checked and it doesn’t seem to use your custom backward pass. it uses pytorches default i believe. way to check- try using breakpoints maybe on spyder or jupyter. insert break points inside forward and backward. 2nd compare your loss values for both cuda and cpu.