Is Scale layer available in Pytorch?


#1

I want to scale the feature after normalization, In caffe,Scale can be performed by Scale Layer,
Is Scale layer available in Pytorch?


#2

do you mean like:

scale_factor = 2
y = x * scale_factor

#3

Yes,But Scale weights should be learnable


#4

okay, so:

scale_factor = Variable(torch.Tensor[2], requires_grad=True)
y = x * scale_factor

#5

ok,I will try. It’s would be great that Scale can be performed in a module, so it will be easy for me to control the parameter just like gamma and beta in BatchNorm.
Thank you for your quick reply.


(Danylo Ulianych) #6
class ScaleFunc(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input, scale):
        ctx.save_for_backward(input, scale)
        return input * scale

    @staticmethod
    def backward(ctx, grad_output):
        input, scale = ctx.saved_variables
        return grad_output * scale, torch.mean(grad_output * input)


class ScaleLayer(nn.Module):

    def __init__(self, init_value=1e-3):
        super().__init__()
        self.scale = nn.Parameter(torch.FloatTensor(1).fill_(init_value))

    def forward(self, input):
        return ScaleFunc.apply(input, self.scale)

What I’m uncertain about is what’s the correct gradient update for the scalar: torch.sum(grad_output * input) or torch.mean(grad_output * input)? Both works as long as we use gradient descent optimization.


#7

@dizcza wouldn’t autograd handle the backward pass without explicitly having to write it?


(Danylo Ulianych) #8

@AruniRC
In fact, one can easily use the built-in pytorch functional.

class ScaleLayer(nn.Module):

   def __init__(self, init_value=1e-3):
       super().__init__()
       self.scale = nn.Parameter(torch.FloatTensor([init_value]))

   def forward(self, input):
       return input * self.scale

(I edited the example slightly --@colesbury)