MaxPooling gradientcheck return False

1

I did gradient check on maxpooling2d and got returned False.
The code is as below:

from torch.autograd import Variable
import torch
from torch.autograd import gradcheck
import numpy as np
import torch
import torch.nn.functional as F


if __name__ == '__main__':
    # type maxpool 2d:

    kernel_size = 2
    n_channel = 100
    feature_size = 6
    batch_size = 3
    input = (Variable(torch.FloatTensor(torch.randn(1, n_channel, feature_size, feature_size)), requires_grad=True), kernel_size)
    f_max_pooling = F.max_pool2d
    print gradcheck(f_max_pooling, input, eps=1e-3)

If I set the eps as default, it will always return False, if I set it to 1e-3, it sometimes returns False. Any idea why?

2

The gradcheck can be sensitive to numerical precision. You could e.g. run it with .double() to somewhat alleviate that (I think this is what the test-suite does, too).
In the second part, you may be on the bound of the required precision and depending on the randomness you pass or not.

Best regards

Thomas

1 Like
3

Thanks!
I tried with:
input = (Variable(torch.DoubleTensor(torch.randn(1, n_channel, feature_size, feature_size).double()), requires_grad=True), kernel_size)`

This still sometimes return False.
Could you elaborate on

you may be on the bound of the required precision and depending on the randomness you pass or not?

4

Hi,

The problem is that maxpool is not really differentiable. It is not differentiable at many points (every time the input that is the maximum change).
That means that if your test case hits one of these cases (where two input are very very close), the gradient computed by finite difference will be different from the subgradient given by the backward pass.

Given your eps of 1e-3, the gradient will be wrong if in your input, two values for the same patch have a difference smaller than 1e-3. Since your input is quite large in size and your generate data in a very small range, I guess this happens all the time.

2 Likes
5

Very clear now, thanks!