Weight Decay Implementation


I have reproduced ResNeXt with pytorch on CIFAR and results are always slightly below the original torch implementation .

I have been in contact with the author of the paper and he thinks it might be due to differences with the original ResNet implementation, such as applying weight decay to the biases.

I have gone through the pytorch implementation of the weight decay and I have now some doubts I would like to ask:

  • Is weight decay affecting the batchnorm biases by default?
  • Taking a look into the current implementation in sgd.py, could anyone explain what is line 85 doing?

d_p.add_(weight_decay, p.data)

Thank you in advance!


For 1. yes it looks like weight decay is being applied on the biases as well as the weights.

For 2. It is multiplying p.data by weight_decay then adding it to d_p (the gradients).


# 3x3 matrix of ones
xx = nn.Parameter(torch.from_numpy(np.ones((3,3))))
#Parameter containing:
# 1  1  1
# 1  1  1
# 1  1  1
#[torch.DoubleTensor of size 3x3]

# sum the values just so we can backprop a gradient
y = torch.sum(xx**2)
# gradient of size 3x3
d_p = xx.grad.data
# 2  2  2
# 2  2  2
# 2  2  2
#[torch.DoubleTensor of size 3x3]

# do this operation
weight_decay = 3.0
d_p.add_(weight_decay, xx.data)
# 5  5  5
# 5  5  5
# 5  5  5
#[torch.DoubleTensor of size 3x3]

You see the matrix was ones, the gradient matrix was twos, then we multiplied the data by weight decay (3.0), then added it to the gradient to get fives.

You can see on stackexchange for a further explanation, where you’re essentially doing the following update: wi←wi−η∂E∂wi−ηλwi. The weight decay is that final term


Thank you, really nice explanation :smile:

if you want to filter out weight decay only for biases (i.e. have weight decay for weights, but no weight decay for biases), then you can use the per-parameter optimization options, like described here: http://pytorch.org/docs/optim.html#per-parameter-options

To get the biases out of the model, you can use model.named_parameters()

Here is an example that might help:


I want the opposite (decay everywhere), so the problem is solved :slight_smile: Actually your answer helps me in another project.

Thank you!

Well explained… !!!