What's allowed and not allowed in nn.Module.forward?

1

Hello there!

I’m not entirely sure how nn.Module works in detail, so I wanted to ask one question.
Are there any operations that are not allowed or should not be done in forward in order to make backward work properly?

For example (assuming my data is of shape (n, 10), e.g. n observations 10 features each) will backprop work just fine with such definition of forward function?

class SimpleNet(nn.Module):

    def __init__(self):
        super(SimpleNet, self).__init__()
        self.fc1 = nn.Linear(5, 2)
        self.fc2 = nn.Linear(5, 2)
        self.fc_final = nn.Linear(4, 1)

    def forward(self, x):
        x1_1 = F.relu(self.fc1(x.narrow(1, 0, 5)))
        x1_2 = F.relu(self.fc2(x.narrow(1, 5, 10)))
        return self.fc_final(torch.cat([x1_1, x1_2]).view(-1, 10))

Are there any particular limitations with regard to forward? If there are some, I’d like to ask how to achieve an architecture similar to the one presented in a proper way.

2

Yes, it will work just fine.

Not really. The operations can even be non-differentiable if you don’t want gradients. You should always return some variable through.