How to update optimizer when my network has newly added parameters?

1

Hi, I have a network, which will expand its network structure, (let’s say, deepen its layers when forwarding).
However, my optimizer is initilialzed by optimizer = nn.SGD(net.parameters(), lr=1e-3) in initialization time, which means the parameters is different from the acutal parameters when the network expand its depth. Therefore, I want to ask, is it possible to update my newly added parameters to optimizer, to make it aware of the newly added parameters, so that the optimizer can optimize these newly added parameters.
Thanks

1 Like
2

seems to be exactly what you want

optimizer.add_param_group

e.g.

optimizer.add_param_group({'param':new_module.parameters() })

Be careful.

2 Likes
3

Thanks. yes, it solves the optimizer’s parameters problem.
I still wonder, when I add a new nn.Module in my main network forward function, need I use nn.add_module(new_module) to append the new module to my main module or not?
What does nn.add_module use for?

4

model.add_module(new_module) adds the new module to the model’s list of modules. model.new_module = new_module also adds the module to the model’s list of modules.

If a module is not in the model’s list of modules then model.parameters() and model.state_dict() will skip over that module. Given that you manually add the module’s parameters to the optimiser it probably doesn’t matter that model.parameters() skips that module. But if you want to save the model using model.state_dict() then you don’t want to skip any modules.