Build Custom param groups for optimizer

vision
1

In the optmizer we have param groups methods.

  1. What are the default param groups if we dont give any in the optimizer?

  2. How to build the custom param groups ?

2

Hello,

You could get some references from here.
If you don’t give any to the optimizer, it will raise an error of empty parameter list.

You could pass a list of parameters or a dict of parameters with other hyperparameters (such as lr, momentum …) to the oprimizer.

3

Thanks it looks like
add_param_Groups could be of some help.
Is it possilble to give eg.

Assume we have nn.Sequential( L1,l2,l3,l4,l5)
i want three groups
(L1) ,(l2,l3,l4),(l5)

High level logic i can think about is we may need to build the param dictionary for each layer groups and pass as param to add param groups.
Please help build code if you can
THanks in advance…

4

Sure, I wrote a demo for your use case. I assume the Sequential is considered as Seq in your network.

model = MyNetwork()
optimizer = torch.optim.SGD(
    [
        {'params': model.Seq[0].parameters(), 'lr': 0.1},
        {'params': model.Seq[1:3].parameters(), 'lr': 0.01},
        {'params': model.Seq[4].parameters(), 'lr': 0.001},
    ])
print(optimizer.param_groups)
5 Likes
5

thanks so we have to pass list of param dict where dict represents a group here.
few more things

  1. if there are sub children like in resnet ,how do i check if layer has sub children
    Resnet is made up of multiple layers where each layer itself is a sequence of models.
    so if i send some layer and want to validate if it has children or not ?

  2. If below is my model

Sequential(
(0): Lambda()
(1): Conv2d(1, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2))
(2): ReLU()
(3): Conv2d(8, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
(4): ReLU()
(5): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
(6): ReLU()
(7): Conv2d(32, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
(8): ReLU()
(9): AdaptiveAvgPool2d(output_size=1)
(10): Lambda()
(11): Linear(in_features=32, out_features=10, bias=False)
)
Is there a way to check by name of layer that i have reached adaptiveavgpool2d
i intend to form group dynamically ,user will give some indices & by default if no indices given then default groups would be Input and all layers after input till last classification layer .If i give indices then three group minimum

  1. input
  2. after input to till a layer before Avg layer
  3. from Avg to last layer

some more peace of excercise for you :slight_smile:

6

If you want to check a specific layer by name in Sequential, you could try on this:

for module in model.children():
    if isinstance(module, nn.AdaptiveAvgPool2d):
        # your use case

There are many ways to check modules, params and buffers in network, you could check them on docs and play with them to meet your use case.

7

Hi, do you think the order of param matters?

8

I understand how we can split parameters into groups layer-wise, but what if we want to hand pick e.g some parameters from the 1st layer, some from the 2nd, etc. I am having trouble doing that. I have a dictionary in the same form as the network parameters with True or False flags for each parameter. I want to group all those parameters whose corresponding flag is true together and those with a false flag together in a different group. Any ideas on how to do that? Thank you in advance for any suggestions!