Hello all, I have read implementation of densenet in torch vision example. I found that they set bias=False in convolution after batch norm layer. It looks redundance because the batch norm did it. Do we have any reason why they add it? Thanks
Could you explain your concern a bit?
Usually the bias
is removed in conv layers before a batch norm layer, as the batch norm’s beta
parameter (bias
of nn.BatchNorm
) will have the same effect and the bias
of the conv layer might be canceled out by the mean subtraction.
From the batch norm paper:
Note that, since we normalize Wu+b, the bias b can be ignored since its effect will be canceled by the subsequent mean subtraction (the role of the bias is subsumed by β in Alg. 1).
Sorry i mistaken. Bias= false means does not set bias and default is True. Forget it. Thanks so much
I guess if we add bias=True
on layers after batch_norm layers then there shouldn’t be any issue right? At least that’s what I understand from the last layers of torchvision.models.vgg16_bn()
:
(avgpool): AdaptiveAvgPool2d(output_size=(7, 7))
(classifier): Sequential(
(0): Linear(in_features=25088, out_features=4096, bias=True)
(1): ReLU(inplace=True)
(2): Dropout(p=0.5, inplace=False)
(3): Linear(in_features=4096, out_features=4096, bias=True)
(4): Dropout(p=0.5, inplace=False)
(5): Linear(in_features=4096, out_features=1024, bias=True)
)
There shouldn’t be any error regarding the code. However, as mentioned above you might save the computation in specific use cases.