I want to scale the feature after normalization, In caffe,Scale can be performed by Scale Layer,
Is Scale layer available in Pytorch?
2 Likes
do you mean like:
scale_factor = 2
y = x * scale_factor
Yes,But Scale weights should be learnable
okay, so:
scale_factor = Variable(torch.Tensor[2], requires_grad=True)
y = x * scale_factor
ok,I will try. It’s would be great that Scale can be performed in a module, so it will be easy for me to control the parameter just like gamma and beta in BatchNorm.
Thank you for your quick reply.
1 Like
class ScaleFunc(torch.autograd.Function):
@staticmethod
def forward(ctx, input, scale):
ctx.save_for_backward(input, scale)
return input * scale
@staticmethod
def backward(ctx, grad_output):
input, scale = ctx.saved_variables
return grad_output * scale, torch.mean(grad_output * input)
class ScaleLayer(nn.Module):
def __init__(self, init_value=1e-3):
super().__init__()
self.scale = nn.Parameter(torch.FloatTensor(1).fill_(init_value))
def forward(self, input):
return ScaleFunc.apply(input, self.scale)
What I’m uncertain about is what’s the correct gradient update for the scalar: torch.sum(grad_output * input)
or torch.mean(grad_output * input)
? Both works as long as we use gradient descent optimization.
1 Like
@AruniRC
In fact, one can easily use the built-in pytorch functional.
class ScaleLayer(nn.Module):
def __init__(self, init_value=1e-3):
super().__init__()
self.scale = nn.Parameter(torch.FloatTensor([init_value]))
def forward(self, input):
return input * self.scale
(I edited the example slightly --@colesbury)
2 Likes