Will parameters get updated if they are not defined in '__init__'

class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, 3, padding=1)
        self.relu1 = nn.ReLU(True)
        self.conv2 = nn.Conv2d(64, 1, 3, padding=1)

    def forward(self, input):
        return self.conv2(self.relu1(self.conv1))

class MyNet2(nn.Module):
    def __init__(self):
        super(MyNet2, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, 3, padding=1)
        self.relu1 = nn.ReLU(True)
     #  move this to forward
     #  self.conv2 = nn.Conv2d(64, 1, 3, padding=1)

    def forward(self, input):
        conv2_filters = torch.randn(1, 64, 3, 3)
        return F.conv2d(self.relu1(self.conv1)), conv2_filters, padding=1)

class MyNet3(nn.Module):
    def __init__(self):
        super(MyNet3, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, 3, padding=1)
        self.relu1 = nn.ReLU(True)
     #  move this to forward
     #  self.conv2 = nn.Conv2d(64, 1, 3, padding=1)

    def forward(self, input):
        self.conv2_filters = torch.randn(1, 64, 3, 3)
        return F.conv2d(self.relu1(self.conv1)), self.conv2_filters, padding=1)

I suspect that optimizer can not get the parameters of conv2 in MyNet2 and MyNet3, then we must define the parameters in __init__ if we want them to get updated in optimizer.step() ?

The point is not that they have to be registered during __init__ but before you create the optimizer since references to the parameters are given to. The optimizer.

For example you could have a function build_model which creates the layers etc. and as long as you call it before creating the optimizers, you should be fine.

Note: when using functional functions like conv2d, you also should register weights and bias as torch.nn.Parameter

Thank you, you are totally correct!

Hi,

How can the params of conv2 register to optimizer in Net2 and Net3 in your post above?
I am really confused about it.

Hi,
In Net2 and Net3, they are not registered to optimizer. They are not Parameters at all. Only when they are registered to Parameters before calling optim.SGD(model.parameters(), lr = 0.01, momentum=0.9), then it get updated when running optimizer.step().
Even for MyNet4, conv2_filters are not updated as usually we define the parameters to be updated right after the definition of the network.

class MyNet4(nn.Module):
    def __init__(self):
        super(MyNet3, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, 3, padding=1)
        self.relu1 = nn.ReLU(True)
     #  move this to forward
     #  self.conv2 = nn.Conv2d(64, 1, 3, padding=1)

    def forward(self, input):
        # I change its type to Parameter
        self.conv2_filters = nn.Parameter(torch.randn(1, 64, 3, 3))
        return F.conv2d(self.relu1(self.conv1)), self.conv2_filters, padding=1)

Hi, thanks for your reply.

I just confused about the topic, I tried register conv2_filters to opt by model.parameters() but I can’t. So how can we add conv2_filters parameter group to opt we defined then we can update its weights when running opt.step()