Any purpose to set bias=False in DenseNet torchvision?

Hello all, I have read implementation of densenet in torch vision example. I found that they set bias=False in convolution after batch norm layer. It looks redundance because the batch norm did it. Do we have any reason why they add it? Thanks

1 Like

Could you explain your concern a bit?

Usually the bias is removed in conv layers before a batch norm layer, as the batch norm’s beta parameter (bias of nn.BatchNorm) will have the same effect and the bias of the conv layer might be canceled out by the mean subtraction.

From the batch norm paper:

Note that, since we normalize Wu+b, the bias b can be ignored since its effect will be canceled by the subsequent mean subtraction (the role of the bias is subsumed by β in Alg. 1).

5 Likes

Sorry i mistaken. Bias= false means does not set bias and default is True. Forget it. Thanks so much

I guess if we add bias=True on layers after batch_norm layers then there shouldn’t be any issue right? At least that’s what I understand from the last layers of torchvision.models.vgg16_bn():

(avgpool): AdaptiveAvgPool2d(output_size=(7, 7))
(classifier): Sequential(
(0): Linear(in_features=25088, out_features=4096, bias=True)
(1): ReLU(inplace=True)
(2): Dropout(p=0.5, inplace=False)
(3): Linear(in_features=4096, out_features=4096, bias=True)
(4): Dropout(p=0.5, inplace=False)
(5): Linear(in_features=4096, out_features=1024, bias=True)
)

There shouldn’t be any error regarding the code. However, as mentioned above you might save the computation in specific use cases.