Hello all, I have read implementation of densenet in torch vision example. I found that they set bias=False in convolution after batch norm layer. It looks redundance because the batch norm did it. Do we have any reason why they add it? Thanks

Could you explain your concern a bit?

Usually the `bias`

is removed in conv layers before a batch norm layer, as the batch normâ€™s `beta`

parameter (`bias`

of `nn.BatchNorm`

) will have the same effect and the `bias`

of the conv layer might be canceled out by the mean subtraction.

From the batch norm paper:

Note that, since we normalize Wu+b, the bias b can be ignored since its effect will be canceled by the subsequent mean subtraction (the role of the bias is subsumed by Î˛ in Alg. 1).

Sorry i mistaken. Bias= false means does not set bias and default is True. Forget it. Thanks so much

I guess if we add `bias=True`

on layers after batch_norm layers then there shouldnâ€™t be any issue right? At least thatâ€™s what I understand from the last layers of `torchvision.models.vgg16_bn()`

:

(avgpool): AdaptiveAvgPool2d(output_size=(7, 7))

(classifier): Sequential(

(0): Linear(in_features=25088, out_features=4096, bias=True)

(1): ReLU(inplace=True)

(2): Dropout(p=0.5, inplace=False)

(3): Linear(in_features=4096, out_features=4096, bias=True)

(4): Dropout(p=0.5, inplace=False)

(5): Linear(in_features=4096, out_features=1024, bias=True)

)

There shouldnâ€™t be any error regarding the code. However, as mentioned above you might save the computation in specific use cases.