Bias-only layer

1

This question was asked about 5 years ago but I feel like it might be appropriate to revisit it:

Can we build a learnable bias layer that will work for arbitrary-sized input? (E.g., a single parameter that is added to every element of a scalar or an image or a video, depending on input tensor shape.)

Is the answer just to build a trainable parameter and broadcast it?

2

Yes, your idea to create a custom layer with the trainable bias (containing a single value) and broadcast it to the input shape sounds valid.

1 Like
3

Thanks. It does seem to work as expected:

class BiasLayer(torch.nn.Module):
    def __init__(self) -> None:
        super().__init__()
        bias_value = torch.randn((1))
        self.bias_layer = torch.nn.Parameter(bias_value)
    
    def forward(self, x):
        return x + self.bias_layer