# Is Scale layer available in Pytorch？

**URL:** <https://discuss.pytorch.org/t/is-scale-layer-available-in-pytorch/7954>\
**Category:** Uncategorized\
**Created:** [September 28, 2017, 2:25am UTC](https://discuss.pytorch.org/t/is-scale-layer-available-in-pytorch/7954 "2017-09-28T02:25:37Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![JiangFeng](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ecccb3/32.png) [@JiangFeng](https://discuss.pytorch.org/u/JiangFeng)\
**Post date:** [September 28, 2017, 2:25am UTC](https://discuss.pytorch.org/t/is-scale-layer-available-in-pytorch/7954/1 "2017-09-28T02:25:37Z")

</div>

I want to scale the feature after normalization， In caffe，Scale can be performed by Scale Layer,  
Is Scale layer available in Pytorch？

---

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [September 28, 2017, 2:27am UTC](https://discuss.pytorch.org/t/is-scale-layer-available-in-pytorch/7954/2 "2017-09-28T02:27:47Z")

</div>

do you mean like:

```auto
scale_factor = 2
y = x * scale_factor

```

---

<div class="post-metadata">

**Author:** ![JiangFeng](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ecccb3/32.png) [@JiangFeng](https://discuss.pytorch.org/u/JiangFeng)\
**Post date:** [September 28, 2017, 2:31am UTC](https://discuss.pytorch.org/t/is-scale-layer-available-in-pytorch/7954/3 "2017-09-28T02:31:09Z")

</div>

Yes，But Scale weights should be learnable

---

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [September 28, 2017, 2:31am UTC](https://discuss.pytorch.org/t/is-scale-layer-available-in-pytorch/7954/4 "2017-09-28T02:31:55Z")

</div>

okay, so:

```python
scale_factor = Variable(torch.Tensor[2], requires_grad=True)
y = x * scale_factor

```

---

<div class="post-metadata">

**Author:** ![JiangFeng](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ecccb3/32.png) [@JiangFeng](https://discuss.pytorch.org/u/JiangFeng)\
**Post date:** [September 28, 2017, 2:46am UTC](https://discuss.pytorch.org/t/is-scale-layer-available-in-pytorch/7954/5 "2017-09-28T02:46:16Z")

</div>

ok，I will try. It’s would be great that Scale can be performed in a module, so it will be easy for me to control the parameter just like gamma and beta in BatchNorm.  
Thank you for your quick reply.

---

<div class="post-metadata">

**Author:** ![dizcza](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dizcza/32/14730_2.png) [@dizcza](https://discuss.pytorch.org/u/dizcza)\
**Post date:** [January 25, 2018, 3:31pm UTC](https://discuss.pytorch.org/t/is-scale-layer-available-in-pytorch/7954/6 "2018-01-25T15:31:20Z")

</div>

```auto
class ScaleFunc(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input, scale):
        ctx.save_for_backward(input, scale)
        return input * scale

    @staticmethod
    def backward(ctx, grad_output):
        input, scale = ctx.saved_variables
        return grad_output * scale, torch.mean(grad_output * input)

class ScaleLayer(nn.Module):

    def __init__ (self, init_value=1e-3):
        super(). __init__ ()
        self.scale = nn.Parameter(torch.FloatTensor(1).fill_(init_value))

    def forward(self, input):
        return ScaleFunc.apply(input, self.scale)

```

What I’m uncertain about is what’s the correct gradient update for the scalar: `torch.sum(grad_output * input)` or `torch.mean(grad_output * input)`? Both works as long as we use gradient descent optimization.

---

<div class="post-metadata">

**Author:** ![AruniRC](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arunirc/32/1695_2.png) [@AruniRC](https://discuss.pytorch.org/u/AruniRC)\
**Post date:** [February 13, 2018, 2:22pm UTC](https://discuss.pytorch.org/t/is-scale-layer-available-in-pytorch/7954/7 "2018-02-13T14:22:14Z")

</div>

@dizcza wouldn’t autograd handle the backward pass without explicitly having to write it?

---

<div class="post-metadata">

**Author:** ![dizcza](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dizcza/32/14730_2.png) [@dizcza](https://discuss.pytorch.org/u/dizcza)\
**Post date:** [February 13, 2018, 3:41pm UTC](https://discuss.pytorch.org/t/is-scale-layer-available-in-pytorch/7954/8 "2018-02-13T15:41:11Z")

</div>

@AruniRC  
In fact, one can easily use the built-in pytorch functional.

```auto
class ScaleLayer(nn.Module):

   def __init__ (self, init_value=1e-3):
       super(). __init__ ()
       self.scale = nn.Parameter(torch.FloatTensor([init_value]))

   def forward(self, input):
       return input * self.scale

```

(I edited the example slightly --@colesbury)