Question about the input x of the forward pass (nn.module)

data
1

Hi guys,
Total noob here.

So, I have the following code

class MyModel(nn.Module):
    def __init__(self, depth, some var, some var1):
         super(MyModel self).__init__()

  def L1_block(self, x, before_trans = True):
    some code

  def L2_block(self, x, before_trans = True):
    some code

 def U1_block(self, x, before_trans = True):
    some code

 def U2_block(self, x, before_trans = True):
    some code

  def forward(self, x, concat_only=True):
        output_var = ['l1_out', 'l2_out', 'u1_out', 'u2_out']
        output_var_t = ['l1_out_t', 'l2_out_t', 'u1_out_t', 'u2_out_t']
        l1_out = self.L1_block(x, before_trans=True)
        l1_out_t = self.L1_block(x, before_trans=False)
        l2_out = self.L2_block(l1_out_t, before_trans=True)
        l2_out_t = self.L2_block(l1_out_t, before_trans=False)
        u1_out = self.U1_block(l2_out_t, l1_out, l2_out, x, before_trans=True)
        u1_out_t = self.U1_block(l2_out_t, l1_out, l2_out, x, before_trans=False)
        u2_out = self.U2_block(u1_out_t, l1_out, l2_out, u1_out, x, before_trans=True)
        u2_out_t = self.U2_block(u1_out_t, l1_out, l2_out, u1_out, x, before_trans=False)
        if concat_only:
            return dict(zip(output_var, (l1_out, l2_out, u1_out, u2_out)))
        else:
            return dict(zip(output_var_t, (l1_out_t, l2_out_t, u1_out_t, u2_out_t)))

My question is, in the forward pass, will the x be the same all through, or will it change at each line of code? x is the input data (Iā€™d assume). I want the x, in this case, to be the same all through the forward, hence the reason for calling it at each block of code.
Thanks so much for the help.

2

Yes, x is the passed input data and will be mapped to input in this example:

output = model(input)

It depends on the used modules if x stays the same or will be modified.
I.e. if any module uses the x input in an inplace operation as the first operation, x would be manipulated.
Here is a small example:

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.module1 = nn.Linear(1, 1)
        self.module2 = nn.ReLU()
        self.module3 = nn.ReLU(inplace=True)
        
    def forward(self, x):
        print(x) # original input
        out1 = self.module1(x)
        print(x) # unmodified
        out2 = self.module2(x)
        print(x) # unmodified
        out3 = self.module3(x)
        print(x) # modified!
        return out3
    
model = MyModel()
x = torch.tensor([[-0.5], [1.0]])
print(x)
# tensor([[-0.5000],
#        [ 1.0000]])

out = model(x)
# tensor([[-0.5000],
#         [ 1.0000]])
# tensor([[-0.5000],
#         [ 1.0000]])
# tensor([[-0.5000],
#         [ 1.0000]])
# tensor([[0.],
#         [1.]])
3

hi ptrblck,

Thanks for your swift response.

what if an inplace operation was part of the function I used in the forward pass?

say for example an in-place operation as part of the functions of the class I used in my Model class. Will this affect the input. if yes how can I overturn this in my forward pass?

I am sorry if my question sounds amateur.
Thank you.

4

I think this would match my example in self.module3 and will thus change the input x or what would the difference be?

You could clone() the tensor before passing it to the module, which would create a copy (thus use more memory) but would avoid the inplace manipulation on the original tensor.

5

Thanks once again ptrblck!

so, about cloning, do you advise I clone the input in the function of the class or in Mymodel class? Which is faster?

class MyFunctionclass(nn.Module):
   def __init__(self, somevar, somevar2)
       super(MyFunctionclass, self).__init__()
   
   def forward(self, x):
     x_ = x.clone()
     some code

Or

class MyModel(nn.Module):
    def __init__(self, depth, some var, some var1):
         super(MyModel self).__init__()

  def L1_block(self, x, before_trans = True):
    some code

  def L2_block(self, x, before_trans = True):
    some code

 def U1_block(self, x, before_trans = True):
    some code

 def U2_block(self, x, before_trans = True):
    some code

  def forward(self, x, concat_only=True):
        x_ = x.clone()
        output_var = ['l1_out', 'l2_out', 'u1_out', 'u2_out']
        output_var_t = ['l1_out_t', 'l2_out_t', 'u1_out_t', 'u2_out_t']
        l1_out = self.L1_block(x_, before_trans=True)
        l1_out_t = self.L1_block(x_, before_trans=False)
        l2_out = self.L2_block(l1_out_t, before_trans=True)
        l2_out_t = self.L2_block(l1_out_t, before_trans=False)
        u1_out = self.U1_block(l2_out_t, l1_out, l2_out, x_, before_trans=True)
        u1_out_t = self.U1_block(l2_out_t, l1_out, l2_out, x_, before_trans=False)
        u2_out = self.U2_block(u1_out_t, l1_out, l2_out, u1_out, x_, before_trans=True)
        u2_out_t = self.U2_block(u1_out_t, l1_out, l2_out, u1_out, x_, before_trans=False)
        if concat_only:
            return dict(zip(output_var, (l1_out, l2_out, u1_out, u2_out)))
        else:
            return dict(zip(output_var_t, (l1_out_t, l2_out_t, u1_out_t, u2_out_t)))
6

The second approach could still hit the issue as you are reusing the same x_. If self.L1_block applies an inplace operation to x_ the next usage could be wrong.
This would make sure to pass the original inputs to each module:

def forward(self, x):
    out1 = self.layer1(x.clone())
    out2 = self.layer2(x.clone())

Note that this will add an overhead since copies are now created, but would be safe from the functional perspective in case you are unsure if any module uses an inplace op as the first operation.

1 Like
7

thank you for the help!