Suggest to add more tips in F.grid_sample() function

vision
1

Hi,

I notice that in the newest version Pytorch, when query the coords in the right side and bottom side, it will assign the values of the padding part (align_corners=False). For example, using the example codes provided in https://discuss.pytorch.org/t/solved-torch-grid-sample/51662/2?u=windvchen, the results will be as following when align_corners=False:

input = torch.arange(4*4).view(1, 1, 4, 4).float()
print(input)
> tensor([[[[ 0.,  1.,  2.,  3.],
          [ 4.,  5.,  6.,  7.],
          [ 8.,  9., 10., 11.],
          [12., 13., 14., 15.]]]])

# Create grid to upsample input
d = torch.linspace(-1, 1, 8)
meshx, meshy = torch.meshgrid((d, d))
grid = torch.stack((meshy, meshx), 2)
grid = grid.unsqueeze(0) # add batch dim

output = torch.nn.functional.grid_sample(input, grid, mode='bilinear', align_corners=False)
print(output)
> tensor([[[[ 0.0000,  0.0357,  0.3214,  0.6071,  0.8929,  1.1786,  1.4643,
            0.7500],
          [ 0.1429,  0.3571,  0.9286,  1.5000,  2.0714,  2.6429,  3.2143,
            1.6429],
          [ 1.2857,  2.6429,  3.2143,  3.7857,  4.3571,  4.9286,  5.5000,
            2.7857],
          [ 2.4286,  4.9286,  5.5000,  6.0714,  6.6429,  7.2143,  7.7857,
            3.9286],
          [ 3.5714,  7.2143,  7.7857,  8.3571,  8.9286,  9.5000, 10.0714,
            5.0714],
          [ 4.7143,  9.5000, 10.0714, 10.6429, 11.2143, 11.7857, 12.3571,
            6.2143],
          [ 5.8571, 11.7857, 12.3571, 12.9286, 13.5000, 14.0714, 14.6429,
            7.3571],
          [ 3.0000,  6.0357,  6.3214,  6.6071,  6.8929,  7.1786,  7.4643,
            3.7500]]]])

It can be seen that the results are totally not what we want! While if we add the padding_mode=‘border’, the results seem to be right.

> tensor([[[[ 0.0000,  0.0714,  0.6429,  1.2143,  1.7857,  2.3571,  2.9286,
            3.0000],
          [ 0.2857,  0.3571,  0.9286,  1.5000,  2.0714,  2.6429,  3.2143,
            3.2857],
          [ 2.5714,  2.6429,  3.2143,  3.7857,  4.3571,  4.9286,  5.5000,
            5.5714],
          [ 4.8571,  4.9286,  5.5000,  6.0714,  6.6429,  7.2143,  7.7857,
            7.8571],
          [ 7.1429,  7.2143,  7.7857,  8.3571,  8.9286,  9.5000, 10.0714,
           10.1429],
          [ 9.4286,  9.5000, 10.0714, 10.6429, 11.2143, 11.7857, 12.3571,
           12.4286],
          [11.7143, 11.7857, 12.3571, 12.9286, 13.5000, 14.0714, 14.6429,
           14.7143],
          [12.0000, 12.0714, 12.6429, 13.2143, 13.7857, 14.3571, 14.9286,
           15.0000]]]])

This phenomenon can also be observed when change the mode to “nearest”. I guess what lead to this may be that when set align_corners=False, the current Pytorch tends to take the right and bottom padding part as boundary. Thus the padding_type “border” must be used together to ensure right results.

I think it is a nontrivial problem that users should pay attention. However, in the current Pytorch, there seem to miss the corresponding annotations to prompt users.

2

I see a warning executing my code snippet from the older post given as:

UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.

which results in a different output than seen in my old post.
Setting align_corners=True to use the old behavior results in the same outputs again.

Your approach of using align_corners=False with padding_mode="border" does not yield the same result.

3

Hi there, thanks for your reply. Maybe I didn’t describe my questions clearly enough before.

I just doubt about the results when set the align_corners=False. Take a more simple example, the input is:

tensor([[[[ 0.,  1.,  2.,  3.],
          [ 4.,  5.,  6.,  7.],
          [ 8.,  9., 10., 11.],
          [12., 13., 14., 15.]]]])

When I set align_corners=False and mode=‘nearest’, after using a 8 X 8 coordinates grid map to query the input, the results should be expected to like this:

tensor([[[[ 0.,  0.,  1.,  1.,  2.,  2.,  3.,  3.],
          [ 0.,  0.,  1.,  1.,  2.,  2.,  3.,  3.],
          [ 4.,  4.,  5.,  5.,  6.,  6.,  7.,  7.],
          [ 4.,  4.,  5.,  5.,  6.,  6.,  7.,  7.],
          [ 8.,  8.,  9.,  9., 10., 10., 11., 11.],
          [ 8.,  8.,  9.,  9., 10., 10., 11., 11.],
          [12., 12., 13., 13., 14., 14., 15., 15.],
          [12., 12., 13., 13., 14., 14., 15., 15.]]]])

i.e., upsample 2x size. However, when execute the codeline:

torch.nn.functional.grid_sample(input, grid, mode='nearest', align_corners=False)

What we can get seems not we want, the bottom line and right column values are set to zero:

tensor([[[[ 0.,  0.,  1.,  1.,  2.,  2.,  3.,  0.],
          [ 0.,  0.,  1.,  1.,  2.,  2.,  3.,  0.],
          [ 4.,  4.,  5.,  5.,  6.,  6.,  7.,  0.],
          [ 4.,  4.,  5.,  5.,  6.,  6.,  7.,  0.],
          [ 8.,  8.,  9.,  9., 10., 10., 11.,  0.],
          [ 8.,  8.,  9.,  9., 10., 10., 11.,  0.],
          [12., 12., 13., 13., 14., 14., 15.,  0.],
          [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]]]])