Why my Batch-Norm Layer parameters doesn't get udates

vision
1

Hi, during the training process of a ResNet model I have modified the affine parameters just before the loss.back as below:

    i = 0
    for layer in model.modules():
        if isinstance(layer, nn.BatchNorm2d):
            layer.bias = nn.Parameter(value 1)
            layer.weight = nn.Parameter(value 2)
            i += 1

I realized the layer.bias and layer.weight does not get updated during the back-prob and remains as 0 and 1 respectively while req_grad is true and the optimizer produces the appropriate gradient tensor. I am wondering if these assignment has caused the problem? If yes, is there an alternative approach?

2
  • you would want to keep the Parameters the same thing after you instantiated the optimizer.
  • if what you want is “set the parameter to something”, try with torch.no_grad(): layer.weight.copy_(value 1). This copies the value 1 to the parameter rather than replacing the parameter with a completely unrelated one the optimizer doesn’t know anything about.

Would that make sense?

Best regards

Thomas

3

Thanks, @tom, I got the issue.