Give different lr for different parameters in the same layer

Hi, let’s say I have a convolution layer with weights of size: [64,64,3,3]. now I want to give different lrs , acctualy I want to freeze part of the kernel (for pure research), I first tried:

model.layer_name.weight[1:,:,:,:].requires_grad  = False

This returned:

RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().     

Furthermore, I tried to think how to use the normal format I’m using to define different lr for different layers:

    parameters= []
    ft_module_names = ['layername']
    for k, v in model.named_parameters():
        for ft_module in ft_module_names:
            if ft_module == k:
               parameters.append({'params': v, 'lr':args.lr_new})
             parameters.append({'params': v})
    optimizer = torch.optim.SGD(parameters,,

But I wasn’t able to think how to modify it so that it will be compatible for different lrs in the same layers

I’ll be happy if you have any ideas!

ping, any idea someone?

As the error suggests, you won’t be able to change the learning rate of a specific part of the parameter.
You could register a hook to this parameter (param.register_hook) and manipulate the gradients by scaling them with your learning rates.