Why is .clone() necesary after expand

Hello all,

as doc of clone function shows

This function is differentiable, so gradients will flow back from the result of this operation to input . To create a tensor without an autograd relationship to input see detach().

for instance,

src = torch.rand(3, 5, requires_grad =True)
dst = src.clone()

The gradients that related to dst tensor will also flow back to src tensor. As so far I guess I’m right.

Here is the case where I donot understand why clone is needed.

    def forward(self, image_size, feats, boxes):
        """
        image_size: image size, height, width
        feats:
        boxes:
        """
        B = boxes.size(0)
        print("boxes.size", boxes.size())
        
        # convert box to affine matrix
        self.affine_regressor.set_size(image_size)
        theta = self.affine_regressor.forward(boxes, image_size)
         
        # generate grids
        grid = self.grid_generator.forward(theta).to(feats.device)
        print("grid.size", grid.size())
        
        # expand inputs for bilinear sampling
        print("feats.size", feats.size())
        feats = feats.expand(B, feats.size(0), feats.size(1), feats.size(2))
        # ^^^^^^^^^^ why is clone needed here??^^^^^^^^^^^^^^^^^^^^ 

        # bilinear sampling
        box_features = self.roi_sampler.forward(feats, grid)
        return box_features

The function convert detected boxes Bx4 into Bx2x3 normalized matrixes, then using these matrixes to generate grids for bilinear sampling, as shown at the last line. The definition of roi_sampler can be located BilinearSamplerBHWD.cu.

After bilinear sampling, I visualized the result, as the following image shows,

the cats in each box are covered by unexpected colors. If I add a clone() after expanding the input feats (3, H, W) to (B, 3, H, W). I get the correcte results, say in each box, the cat is interpolated with given expanded feats w.r.t. the generated grids (feats are outputs of backbone model, here VGG16), as depicted in the following figure.

My question:

  • After expanding, the feas sizes with or without clone are the same 6x3xHxW. What kind of role is played by the clone function. The feats are already expanded in the correct dims.

  • With clone(), the gradients will flow back to the expanded tensor (B, 3, H, W), which are originally based on (3, H, W). Do the gradients flow back further to this base tensor. In my case, I need the gradients of the base tensor.

Thanks.

Hi,

What .clone() does is allocate brand new memory and copy the content of the Tensor into it. It always creates a new contiguous Tensor as output.
The thing with expand is that it does not actually allocate new memory but play with stride. Unfortunately, not all functions support non-standard strides (non-contiguous Tensors).
I guess that your function is one of these and so the clone fixes it by making sure the input is contiguous.

A more explicit way to do this, is to do feats = feats.contiguous() so that, if feats is already contiguous, nothing will happen. Compared to clone which will always make a full copy.

Hi albanD,

yes, you are absolutely right. The tensor feats I tested here is dummy tensor, not the direct output of a backbone model. If I put my bilinear sampling module into my complete model. The problem is automatically resolved. Actually, if there is tensor that is non-contiguous, it might trigger runtime error if some ops need the tensor to be contiguous. Thanks for your input.