Leaf variable was used in an inplace operation

technetium-43 · June 7, 2021, 1:13am

Thanks this worked for me. I missed the code before updating the parameters.

with torch.no_grade():

JoshGreifer · June 15, 2021, 11:48am

Substituting

x += 1

with

x = x + 1

will work and achieve the desired effect (at the expense of creating a temporary and doing an assignment)

Linga_Abhishek · September 24, 2021, 5:38pm

Hi, thank you for the detailed reply. However, there is no concrete definition for leaf variables than the one you’ve provided here. In fact, the documentation link you provided says that leaf variables are the tensors which have requires_grad set to False.
This means that the documentation is wrong. (I checked as well)
Please correct the documentation if i’m right.

bpfrd · February 8, 2023, 3:29pm

In pixelCNN code (here for example pixelcnn-pytorch/masked_cnn_layer.py at master · axeloh/pixelcnn-pytorch · GitHub) one can multiply the network parameter (which is a leaf node with requires_grad=True) with a mask tensor but still doesn’t get any error! how is this possible?

tom · February 22, 2023, 6:38am

You mean the line self.weight.data *= self.mask below?
So it uses the long-deprecated .data which - similar to the recommended with torch.no_grad(): - disconnects the modification from autograd. This is similar to the optimizer update not being seen by autograd.

Best regards

Thomas

github.com

axeloh/pixelcnn-pytorch/blob/ae5e49424147b7e88d886639df2d1e50335028fa/masked_cnn_layer.py#L17


      
                  super(MaskedConv2d, self).__init__(in_channels, out_channels, kernel_size, padding=padding)
                  # self.register_buffer('mask', self.weight.data.clone())#torch.ones(out_channels, in_channels, kernel_size, kernel_size))
                  # self.mask.fill_(1)
                  kH, kW = kernel_size, kernel_size
                  self.device = torch.device('cuda') if use_cuda else None
                  self.mask = torch.ones(out_channels, in_channels, kH, kW).to(self.device)
                  self.mask[:, :, kH // 2, kW // 2 + (mask_type == 'B'):] = 0
                  self.mask[:, :, kH // 2 + 1:] = 0
          
          
    def forward(self, x):
                  self.weight.data *= self.mask
                  return super(MaskedConv2d, self).forward(x)
          
          

          
class ResidualMaskedConv2d(nn.Module):
              """Residual Masked Conv. layer"""
              def __init__(self, input_dim):
                  super().__init__()
          
          
        self.net = nn.Sequential(
                      MaskedConv2d('B', input_dim, input_dim // 2, kernel_size=1),

Jack88 · May 27, 2024, 1:19pm

leaf tensor could be either, you could use .is_leaf to check if it is a leaf tensor:

attribute require_grad = True but directly create by user
any tensor with require_grad = False