Cosine similarity of model weights of two different checkpoints

vision
1

Hello, I want to analyze the changes in model weights over the training phase. For doing so, I want to use cosine similarity between model weights in two different checkpoints. I was wondering how I could do that? Thanks a lot in advance.

2

You can try something like this. You only need to define a variable to hold the old values and compare to the new ones and after the comparison update the “new” old values.

cos = torch.nn.CosineSimilarity(dim=0)
params = []

for param in model.parameters():
    params.append(param.view(-1))

params = torch.cat(params)

with torch.no_grad():
    # Same input twice --> returns 1
    print(cos(params, params))

    # Small deviation from the original --> close to 1, but smaller
    print(cos(params, params+(torch.rand(params.shape)/100) ))

    # Some random value --> close to 0
    print(cos(params, torch.rand(params.shape)))

Hope this helps :smile:

1 Like