Why there is still a small change in frozen layer parameters?

Hi, there. I meet a strange problem in my model. I have frozen the layer ‘fc1’ in my model, but the parameters of such a layer show a small change after the training. The following is my code for the layer-freeze functions, and the way I check the parameters.

# how I freeze the layer
def set_freeze_by_layer_names(model, layer_names, freeze=True):
    if not isinstance(layer_names, Iterable):
        layer_names = [layer_names]
    for name, child in model.module.named_children():
        if name not in layer_names:
            continue
        print("set freeze for layer {} as: {}".format(name, freeze))
        for param in child.parameters():
            param.requires_grad = not freeze

# how I check the parameters
set_freeze_by_layer_names(model,'fc1', freeze=True)
print('before_train', model.module.fc1[0].weight[0, :10])
training(model)
print('after_train', model.module.fc1[0].weight[0, :10])
set_freeze_by_layer_names(model,'fc1', freeze=False)

The following are the print results:

set freeze for layer fc1 as: True
before_train tensor([-0.0137,  0.0024,  0.0084, -0.0138, -0.0077, -0.0053, -0.0130,  0.0047,
         0.0057, -0.0151], device='cuda:0')
after_train tensor([-0.0138,  0.0023,  0.0083, -0.0138, -0.0078, -0.0054, -0.0131,  0.0046,
         0.0056, -0.0151], device='cuda:0')
set freeze for layer fc1 as: False

I get the reason! Refer to this link, I also need to set param.grad=None if we do not want to accumulate the gradients. So the good freeze function should be:

def set_freeze_by_layer_names(model, layer_names, freeze=True):
    if not isinstance(layer_names, Iterable):
        layer_names = [layer_names]
    for name, child in model.module.named_children():
        if name not in layer_names:
            continue
        print("set freeze for layer {} as: {}".format(name, freeze))
        for param in child.parameters():
            param.requires_grad = not freeze
            param.grad = None