How can i perform Global average pooling before the last fully connected layer

1

class ConvNet(nn.Module):
def init(self):
super(ConvNet, self).init()
self.layer1 = nn.Sequential(
nn.Conv1d(1, 16, kernel_size=5, stride=1 , padding=2),
nn.BatchNorm1d(16),
nn.ReLU(),
nn.MaxPool1d(kernel_size=2, stride=2))
self.layer2 = nn.Sequential(
nn.Conv1d(16, 32 ,kernel_size=5, stride=1, padding=2),
nn.BatchNorm1d(32),
nn.ReLU(),
nn.MaxPool1d(kernel_size=2, stride=2))

    self.drop_out = nn.Dropout()
    
    #self.fc1 = nn.Linear(32*25, 1000)
    self.fc2 = nn.Linear(64, 2)

this step is how the data flow through these layers in forward pass

def forward(self, x):
    
    
    out = self.layer1(x)
    out = self.layer2(out)
    #out = F.adaptive_avg_pool2d(x, (1, 1))
    out = F.avg_pool1d(out,1)
    #out = self.layer3(out)
    #out = out.reshape(-1, 12)
    out = self.drop_out(out)
    #out = self.fc1(out)
    
    out = self.fc2(out)
    return out
2

Based on the Network in Network paper global average pooling is described as:

Instead of adding fully connected layers on top of the feature maps, we take the average of each feature map, and the resulting vector is fed directly into the softmax layer. One advantage of global average pooling over the fully connected layers is that it is more native to the convolution structure by enforcing correspondences between feature maps and categories. Thus the feature maps can be easily interpreted as categories confidence maps. Another advantage is that there is no parameter to optimize in the global average pooling thus overfitting is avoided at this layer. Futhermore, global average pooling sums out the spatial information, thus it is more robust to spatial translations of the input.

which can be achieved by using out = out.mean([2, 3]).

However, since you are using linear layers, I’m not sure, if you are really looking for the global average pooling layers.

1 Like
3

i want to remove those fully connected layers and replace them with average pooling layer.

4

In that case the mean operation should work.

5

how will i implement that please. i am searching for it for long time.

6

I’ve given the code in my previous code:

out = out.mean([2, 3])

This will calculate the mean in dim2 and dim2 and return a 2-dimensional tensor, which should then (based on the paper desciption) contain the class logits in the channel dimension (dim1).

7

Dimension out of range (expected to be in range of [-2, 1], but got 2)

8

Oh, my bad. I missed that you are using nn.Conv1d. In that case, you should only calculate the mean in the temporal dimension: out = out.mean(2).
Also, note that you would have to apply it before feeding the activation to your linear layers and before reshaping the tensor.
Based on the error message, it seems you are trying to apply it on a flattened tensor.