Extract Only Batch Norm Parameters

vision
1

Hi. I would like to extract all batch norm parameters from the pre-trained model?
May I know if you have any proper way to form a list of batch norm parameters?
This is due to the reason that I would like to retrain these parameters separately.

# for layer in resnet.modules():
#    if isinstance(layer,torch.nn.modules.batchnorm.BatchNorm2d): 
#        .....
2

Your code should work to check for all batchnorm layers in the model.
Would you like to store the affine parameters only (weight and bias) or also the running estimates?
Where are you stuck at the moment? Are you trying to append all wanted parameters to a list?

3

Hi thank you for the prompt response.

I would like to append all those BN parameters to form a list for separate training with a different learning rate.

Is my implementation reasonable?

bn_paras = list()
for layer in resnet.modules():
    if isinstance(layer,torch.nn.modules.batchnorm.BatchNorm2d): 
        bn_paras.append(layer.parameters())
print(len(bn_paras))
4

Assuming you would like to pass this list to an optimizer, this approach should work:

bn_paras = []
for layer in resnet.modules():
   if isinstance(layer,torch.nn.modules.batchnorm.BatchNorm2d): 
       bn_paras.extend(list(layer.parameters()))
       
optimizer = torch.optim.SGD(bn_paras, lr=1.)
1 Like
5

Thank you very much~~ :")))