Is it possible to name param_group?

1

I am using optimizer.add_param_group() to add parameters to be optimized as and when needed. During training I would like to change the learning rates of different parameter groups in different ways.

One way to do this is to remember the order in which they were added and use that ordering/ID to modify the learning rates.

I was wondering if I can name the param_groups so that I can modify learning rates by accessing them through their names?

6 Likes
2

Very few people know this.

3

Yes, you can. As defined in the Optimizer class:

You can use a dictionary to define a parameter group with the syntax:
param_groups = [{‘params’: COMPONENT_1_PARAMTERS, ‘lr’: COMPONENT_1_LR, ‘name’: ‘GROUP_1’},
{‘params’: COMPONENT_2_PARAMTERS, ‘lr’: COMPONENT_2_LR, ‘name’: ‘GROUP_2’},
…
]

opt.Adam(param_groups, lr=INITIAL_LR)

5 Likes