Can optimizer optimize all the parameters in main net and subnet?

I define the subnet as a Independent network and instantiate it in the main net like this:

Class SubNet(nn.Module):
    def __init__(self):
        ...
    def forward(self, inputs):
        ...

Class MainNet(nn.Module):
    def __init__(self):
        ...
        self.subnet = SubNet()
    def forward(self, inputs):
        ...
        output = self.subnet(feature_map)
        ...

and the optimizer looks like this:

mainnet = MainNet()
optimizer = optim.SGD(mainnet.parameters(), lr=0.01, momentum=0.9)

I just print the parameters number of the main net, I notice that the number changed if I add a subnet in the main net. Does this means that the optimizer will optimize all the parameters in main net and subnet?

Thanks for all the reply.

Hi,I’ve encountered this problem before.
You can use chain to put your model parameters together just following this code:
from itertools import chain
parameters = chain(mainnet.parameters(),subnet.parameters())
optimizer = optim.SGD(parameters, lr=0.01, momentum=0.9)

Thanks for your reply.
But I want to instantiate the subnet in the main net init function, I can’t get the subnet object directly.
Do you mean that I should solve this problem like this:

parameters = chain(mainnet.parameters(), mainnet.subnet.parameters()) ???

To answer your original question, yes, adding a Module as another Module’s attribute will register the former Module as a child of the latter. Parameters of the former will also be parameters of the latter.

Thank you very much! :slight_smile: