How can I compute L2 norm of weight difference?

1

Hello everyone!

I want to check the weight of a specific layer while training ResNet.
And i’ll compute the L2 norm of weight difference.
The formulation is as follows.
image
(where w is the t th weight in ResNet)

How can I calculate this formulation?

2

Here by weight difference you mean the difference in weights of two layers, or difference in weights of the same layer between two different training steps.

3

I mean the difference of the same layer between two different training steps!
difference between weight of t th step and weight of t - 1 th step.

4

So for this you first need to access the weights of a certain layer, this can be done using:

import torch
from torchvision import models
import torch.nn as nn

model = models.resnet18()

for name, param in model.named_parameters():
  print(name)
  print(param)

The above script prints all the layer names and the weights. To store a certain weight layer you can use:

for name, param in model.named_parameters():
  if name=="layer1.0.conv1.weight": #replace the layer name with layer name of your choice
    weight1 = param.detach().cpu().numpy()

This stores the weight parameters as a numpy array in the variable weight1. Between two training steps, you can use the code snippet to get weights for the same layers, and then compute the weight difference, and L2 norm easily.

1 Like
5

Thank you very much. I’ll try this!