MaxPool2d non-deterministic?

1

I have some conv nn and set np.random.seed manually, based on which I later fill in my starting weights of conv and fully-connected layers. To have everything deterministic. And it works. But then I added two MaxPool2d layers which I thought should be deterministic but turns out one of them is not. Basically these ar emy conv layers:

self.conv.add_module("conv_1", conv1)
self.conv.add_module("maxpool_1", torch.nn.MaxPool2d(2)) # 1
self.conv.add_module("relu_1", torch.nn.ReLU())

self.conv.add_module("conv_2", conv2)
self.conv.add_module("relu_2", torch.nn.ReLU())

self.conv.add_module("conv_3", conv3)
self.conv.add_module("maxpool_3", torch.nn.MaxPool2d(2)) # 2
self.conv.add_module("relu_3", torch.nn.ReLU())

I tested it and it turns out that if max pool layer #1 is present, the randomness is not deterministic. Why would that be?

1 Like
2

Do you retrain them or not ?
I have also a determinism problem, so I am following you post just in case.

3

Well yes, these layers are all trainable.

4

I know, but when you add the maxpool, do you retrain the network again ?

5

Hi,

I guess you are using cudnn? If so the default maxpooling algorithm is known not to be deterministic.
You can use torch.backends.cudnn.deterministic = True to force it to use a deterministic algorithm (at a performance cost).

1 Like
6

Thank you albanD a bunch! That was it.
Although I find it weird that a filter as simple as 2x2 max could be non-deterministic -_-

7

If I remember correctly, the computation of the max itself is deterministic, but the backward pass can be implemented much more efficiently if you accept non-deterministic output (which is what cudnn is doing by default).

2 Likes
8

I see.
Anyway, thanks again :).