Mismatch in bias addition

1

Hey Folks,
Recently I came across strange stuff in bias addition. Bias can be added in 2 ways :

  1. Acc = F.conv2d(input, weights , bias) : bias is added inside the convolution layer in PyTorch

  2. Acc = F.conv2d(input, weights , torch.zeros(bias.shape))
    Acc = Acc + bias
    here bias is added externally

Surprisingly I noticed that the Acc after bias addition in the two cases above are different and bias addition in case 2 is more accurate :slight_smile:

Can anyone help me in explaining if PyTorch uses any optimization algorithm while doing bias addition inside a convolutional later?

Thanks in advance

2

Could you explain how you’ve measured the accuracy in both cases and what the ground truth was, please?

3

Can you provide a minimal example which reproduces the problem?

4

Sure,
A1 = F.conv2d(input, weight, torch.zeros_like(bias)) = tensor([128276)]
bias = tensor([16777215])
A1 + bias = tensor([16905492])

A2 = F.conv2d(input, weight, bias) = tensor([16905484])

so A1 is not equals to A2 and A1 is close to the actual addition value

5

Your used numbers are close to the range where integers start to be rounded to multiples of 2 ([2**24, 2**25]) as seen here.
Depending on the order of operations these rounding errors are expected. If you need more precision in this range, you could use float64.

1 Like
6

Thank You @ptrblck .