Calling functions defined in classes from forward

1

My problem is best illustrated with an example.

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()

        self.l1 = nn.Sequential(
            nn.Linear(1200, 1)
        )

    def forward(self, z):
        zh = self.l1(z)
        return x

works. However,

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()

        def l1x(somenumber):
            return nn.Sequential(
                nn.Linear(somenumber, 1)
            )
        self.l1 = l1x

    def forward(self, z):
        zh = self.l1x(1200)(x)
        return x

gives

Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor

when I use cuda. If I edit self.l1x(1200).cuda()(x) this will work too but I already made my generator object cuda() after I initialized the object. Hence I want it to work like everything else without extra steps.

What am I doing wrong?

2

In your second approach you re-create the layer every time you call forward. Are you sure you would like that? Your model won’t learn anything as it’s random in every step.

1 Like
3

No I would not like that. Full disclosure, I am a newbie so haven’t the faintest clue if anything I’m doing makes sense. All I want is to not write the same block over and over again. I think I can define a function that returns the layers within the init, and can call it like

self.l1 = my_func(a, b, c)

but since I’ll call something again under the forward() I feel this is verbose. Or is it necessary and the right way because how pytorch works?

4

The first example should work just fine.
You usual way is to create the layers in __init__ and call them in forward.
If you would like you can also create another member function to create your layers, but you should still call it in __init__.

What do you mean by write the same block over and over again? Do you have an example? Currently your second approach seems to need more lines of code.

5 
class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()

        self.l1 = nn.Sequential(
            nn.Linear(1200, 240)
        )
        self.l2 = nn.Sequential(
            nn.Linear(1200, 100)
        )

    def forward(self, z):
        zh = self.l1(z)
        zh2 = self.l2(z)
        return torch.cat((zh,zh2), dim=1)

Ignore the meaningless network. I am wondering why can’t I define a function

def my_func(n_out):
  return nn.Sequential(
            nn.Linear(1200, n_out)
        )

and call

    def forward(self, z):
        zh = my_func(100)(z)
        zh2 = my_func(200)(z)
        return torch.cat((zh,zh2), dim=1)
6

You could call my_func in __init__ to create your layers:

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()

        self.l1 = self.my_func(100)

    def my_func(self, n_out):
        return nn.Sequential(
            nn.Linear(1200, n_out)
        )

    def forward(self, z):
        z = self.l1(z)
        return z

Inside my_func you are creating the nn.Sequential with the layer inside, i.e. the layer will be created with randomly initialized parameters.
If you call this function in your forward, you will re-create your model in every forward pass.