Batch Normalization of Linear Layers


#1

Is it possible to perform batch normalization in a network that is only linear layers?

For example:

class network(nn.Module):
    def __init__(self):
        super(network, self).__init__()
        self.linear1 = nn.Linear(in_features=40, out_features=320)
        self.linear2 = nn.Linear(in_features=320, out_features=2)

    def forward(input):  # Input is a 1D tensor
        y = F.relu(self.linear1(input))
        # Would it be possible to do a batch normalization of y overhere? If so how?
        y = F.softmax(self.linear2(input))
        return y

#2

Sure! You could just use nn.BatchNorm1d.
There are some minor issues in your code, so here is a working example:

class network(nn.Module):
    def __init__(self):
        super(network, self).__init__()
        self.linear1 = nn.Linear(in_features=40, out_features=320)
        self.bn1 = nn.BatchNorm1d(num_features=320)
        self.linear2 = nn.Linear(in_features=320, out_features=2)

    def forward(self, input):  # Input is a 1D tensor
        y = F.relu(self.bn1(self.linear1(input)))
        y = F.softmax(self.linear2(y), dim=1)
        return y
    
model = network()
x = torch.randn(10, 40)
output = model(x)

You can also put the BatchNorm after the relu, if you like.


#3

@ptrblck I tried that but I received “ValueError: expected a 2D or 3D input (got 1D input).”


#4

Are you sure you are passing your input as [batch_dim, num_features]?
The error sounds like you’ve passed just [num_features] to your model.


#5

When I do that I get a different error.: “ValueError: Expected more than 1 value per channel when training, got input size [1, 320].” This is for Q network, so it only receives one state at a time, hence the batch size of 1.


#6

Then nn.BatchNorm probably won’t work very well.
Have a look at the normalization layers. Maybe LayerNorm or another one will fit your needs.