Output features of conv layers inconsistant after pruning a empty filter

vision
1

Hi~
I encountered a strange phenomenon: after pruning an empty filter in a conv layer, the output features of the conv is inconsistent with the previous one, (The shape change is as expected, what I said is the value inconsistency)
To reproduce:

c1=torch.nn.Conv2d(3,10,7,bias=False)
c2=torch.nn.Conv2d(3,9,7,bias=False)
for i in range(9):
    c1.weight.data[i] = c2.weight.data[i]

c1.weight.data[9]=torch.zeros(3, 7, 7)
data = torch.rand(64,3, 224,224)
out1=c1(data)
out2=c2(data)
print(torch.sum(out1))
print(torch.sum(out2))


tensor(2090009.8750, grad_fn=<SumBackward0>)
tensor(2090010.1250, grad_fn=<SumBackward0>)

The sum of the output features of c1 and c2 are different. c1 only have one more empty filter than c2, and they convolve the same input data, so, the sum of their output features should be the same theoretically. Why the sum of these two output features are different? Is this a bug? or pytorch has some optimization of the convolution operation that results in this inconsistency?

2

The difference is most likely created due to the limited floating point precision and a different order of operations as shown in this example:

x = torch.empty(100, 100, 100).uniform_(0, 100)
print(x.sum())
> tensor(49965816.)
print(x.sum(0).sum(0).sum(0))
> tensor(49965808.)

Also, you shouldn’t use the .data attribute, as it might yield unwanted side effects.

2 Likes