Hi. Today I was trying to convert a model weight to my implementation. I found a problem converting a Squeeze-Excitation Block that is the following:
class SEBlock(nn.Module): def __init__(self, in_channels, out_channels, reduction_ratio=4): super(SEBlock, self).__init__() red_channels = round_by(in_channels / reduction_ratio, 8) self.conv1 = nn.Conv2d(in_channels, red_channels, 1) self.activation = nn.ReLU(inplace=True) self.conv2 = nn.Conv2d(red_channels, out_channels, 1) def forward(self, input): x = F.adaptive_avg_pool2d(input, 1) x = self.conv1(x) x = self.activation(x) x = self.conv2(x) return input * hard_sigmoid(x)
However, in the original model, the pointwise convolutions are
nn.Linear layers. I think that they could be easily converted as such (because the input is a 1x1 image):
linear = nn.Linear(16, 8) conv = nn.Conv2d(16, 8, 1) conv.weight.copy_(linear.weight.reshape(8, 16, 1, 1)); conv.bias.copy_(linear.bias);
Unfortunelly, this does not reproduce the same result!
x = torch.randn((3, 16, 1, 1)) torch.norm(conv(x) - linear(x.view(3, 16))) >>> tensor(16.5712)
What am I missing here?