Hey Folks,
Recently I came across strange stuff in bias addition. Bias can be added in 2 ways :

Acc = F.conv2d(input, weights , bias) : bias is added inside the convolution layer in PyTorch

Acc = F.conv2d(input, weights , torch.zeros(bias.shape))
Acc = Acc + bias
here bias is added externally

Surprisingly I noticed that the Acc after bias addition in the two cases above are different and bias addition in case 2 is more accurate

Can anyone help me in explaining if PyTorch uses any optimization algorithm while doing bias addition inside a convolutional later?

Thanks in advance

ptrblck
November 25, 2022, 8:55pm
#2
Could you explain how you’ve measured the accuracy in both cases and what the ground truth was, please?

jondapper
(John Dapper)
November 26, 2022, 3:44pm
#3
Can you provide a minimal example which reproduces the problem?

Sure,
A1 = F.conv2d(input, weight, torch.zeros_like(bias)) = tensor([128276)]
bias = tensor([16777215])
A1 + bias = tensor([16905492])

A2 = F.conv2d(input, weight, bias) = tensor([16905484])

so A1 is not equals to A2 and A1 is close to the actual addition value

ptrblck
December 2, 2022, 10:35pm
#5
Your used numbers are close to the range where integers start to be rounded to multiples of 2 (`[2**24, 2**25]`

) as seen here .
Depending on the order of operations these rounding errors are expected. If you need more precision in this range, you could use `float64`

.

1 Like