In pytorch nn.conv2d,when the input matrix is big enough,the output is different from sub of matrix input

1 
m = nn.Conv2d(2,4,(1,1))
x=torch.randn(1,2,10,1024)
o1=m(x)
o2=m(x[:,:,:,0:1])
print(o1[:,:,:,0:1].equal(o2))
output:True
x=torch.randn(1,2,10,2000)
o11=m(x)
o22=m(x[:,:,:,0:1])
print(o11[:,:,:,0:1].equal(o22))
output:False
print(o11[:,:,:,0:1].isclose(o22))

why o11[:,:,:,0:1] is different from o22??

2

Using your code I get an abs. error of 1.1921e-07, which is expected for float32 due to its limited floating point precision. These errors are caused by a different order of operation, which might be used depending on the input shape.

3

thanks for your reply！
Is there someway to make o11[:,:,:,0:1] same as o22？
Is it optimized for conv2d when the input size is big?

4

No, I don’t think there is an easy way to create the same errors unless you write an algorithm with a defined sequence of operations (for this slice). Also note that neither of them is “correct” and both show the expected error to the theoretically ground truth value. If you need more precision, you could use float64 instead.

Yes, I guess so, but it would also depend on the used libraries (in this case MKL might be used as the conv is performed on the CPU) as well as the selected algorithm.

5

But I found that the results are same in tensorflow，

input_shape = (10, 10000, 20, 2)
x=tf.random.normal(input_shape)
m=tf.keras.layers.Conv2D(2,(5,1))
y1=m(x)
y2=m(x[:,:,0:1,:])
print(tf.math.reduce_sum(abs(y1[:,:,0:1,:]-y2)))
tf.Tensor(0.0, shape=(), dtype=float32)
6

But I found that the results are same in tensorflow，

input_shape = (10, 10000, 20, 2)
x=tf.random.normal(input_shape)
m=tf.keras.layers.Conv2D(2,(5,1))
y1=m(x)
y2=m(x[:,:,0:1,:])
print(tf.math.reduce_sum(abs(y1[:,:,0:1,:]-y2)))
tf.Tensor(0.0, shape=(), dtype=float32)

l

7

That doesn’t contradict my explanation and maybe TF does not switch between different algorithms (for performance gains) for different input shapes.