Hello all, I have read implementation of densenet in torch vision example. I found that they set bias=False in convolution after batch norm layer. It looks redundance because the batch norm did it. Do we have any reason why they add it? Thanks
Could you explain your concern a bit?
bias is removed in conv layers before a batch norm layer, as the batch norm’s
beta parameter (
nn.BatchNorm) will have the same effect and the
bias of the conv layer might be canceled out by the mean subtraction.
From the batch norm paper:
Note that, since we normalize Wu+b, the bias b can be ignored since its effect will be canceled by the subsequent mean subtraction (the role of the bias is subsumed by β in Alg. 1).
Sorry i mistaken. Bias= false means does not set bias and default is True. Forget it. Thanks so much
I guess if we add
bias=True on layers after batch_norm layers then there shouldn’t be any issue right? At least that’s what I understand from the last layers of
(avgpool): AdaptiveAvgPool2d(output_size=(7, 7))
(0): Linear(in_features=25088, out_features=4096, bias=True)
(2): Dropout(p=0.5, inplace=False)
(3): Linear(in_features=4096, out_features=4096, bias=True)
(4): Dropout(p=0.5, inplace=False)
(5): Linear(in_features=4096, out_features=1024, bias=True)
There shouldn’t be any error regarding the code. However, as mentioned above you might save the computation in specific use cases.