Recently I came across strange stuff in bias addition. Bias can be added in 2 ways :
Acc = F.conv2d(input, weights , bias) : bias is added inside the convolution layer in PyTorch
Acc = F.conv2d(input, weights , torch.zeros(bias.shape))
Acc = Acc + bias
here bias is added externally
Surprisingly I noticed that the Acc after bias addition in the two cases above are different and bias addition in case 2 is more accurate
Can anyone help me in explaining if PyTorch uses any optimization algorithm while doing bias addition inside a convolutional later?
Thanks in advance
Could you explain how you’ve measured the accuracy in both cases and what the ground truth was, please?
Can you provide a minimal example which reproduces the problem?
A1 = F.conv2d(input, weight, torch.zeros_like(bias)) = tensor([128276)]
bias = tensor()
A1 + bias = tensor()
A2 = F.conv2d(input, weight, bias) = tensor()
so A1 is not equals to A2 and A1 is close to the actual addition value
Your used numbers are close to the range where integers start to be rounded to multiples of 2 (
[2**24, 2**25]) as seen here.
Depending on the order of operations these rounding errors are expected. If you need more precision in this range, you could use