Will parameters get updated if they are not defined in '__init__'

autograd
1 
class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, 3, padding=1)
        self.relu1 = nn.ReLU(True)
        self.conv2 = nn.Conv2d(64, 1, 3, padding=1)

    def forward(self, input):
        return self.conv2(self.relu1(self.conv1))

class MyNet2(nn.Module):
    def __init__(self):
        super(MyNet2, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, 3, padding=1)
        self.relu1 = nn.ReLU(True)
     #  move this to forward
     #  self.conv2 = nn.Conv2d(64, 1, 3, padding=1)

    def forward(self, input):
        conv2_filters = torch.randn(1, 64, 3, 3)
        return F.conv2d(self.relu1(self.conv1)), conv2_filters, padding=1)

class MyNet3(nn.Module):
    def __init__(self):
        super(MyNet3, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, 3, padding=1)
        self.relu1 = nn.ReLU(True)
     #  move this to forward
     #  self.conv2 = nn.Conv2d(64, 1, 3, padding=1)

    def forward(self, input):
        self.conv2_filters = torch.randn(1, 64, 3, 3)
        return F.conv2d(self.relu1(self.conv1)), self.conv2_filters, padding=1)

I suspect that optimizer can not get the parameters of conv2 in MyNet2 and MyNet3, then we must define the parameters in __init__ if we want them to get updated in optimizer.step() ?

2

The point is not that they have to be registered during __init__ but before you create the optimizer since references to the parameters are given to. The optimizer.

For example you could have a function build_model which creates the layers etc. and as long as you call it before creating the optimizers, you should be fine.

Note: when using functional functions like conv2d, you also should register weights and bias as torch.nn.Parameter

3

Thank you, you are totally correct!

4

Hi,

How can the params of conv2 register to optimizer in Net2 and Net3 in your post above?
I am really confused about it.

1 Like
5

Hi,
In Net2 and Net3, they are not registered to optimizer. They are not Parameters at all. Only when they are registered to Parameters before calling optim.SGD(model.parameters(), lr = 0.01, momentum=0.9), then it get updated when running optimizer.step().
Even for MyNet4, conv2_filters are not updated as usually we define the parameters to be updated right after the definition of the network.

class MyNet4(nn.Module):
    def __init__(self):
        super(MyNet3, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, 3, padding=1)
        self.relu1 = nn.ReLU(True)
     #  move this to forward
     #  self.conv2 = nn.Conv2d(64, 1, 3, padding=1)

    def forward(self, input):
        # I change its type to Parameter
        self.conv2_filters = nn.Parameter(torch.randn(1, 64, 3, 3))
        return F.conv2d(self.relu1(self.conv1)), self.conv2_filters, padding=1)
6

Hi, thanks for your reply.

I just confused about the topic, I tried register conv2_filters to opt by model.parameters() but I can’t. So how can we add conv2_filters parameter group to opt we defined then we can update its weights when running opt.step()

1 Like