Hello, I want to analyze the changes in model weights over the training phase. For doing so, I want to use cosine similarity between model weights in two different checkpoints. I was wondering how I could do that? Thanks a lot in advance.
You can try something like this. You only need to define a variable to hold the old values and compare to the new ones and after the comparison update the “new” old values.
cos = torch.nn.CosineSimilarity(dim=0) params =  for param in model.parameters(): params.append(param.view(-1)) params = torch.cat(params) with torch.no_grad(): # Same input twice --> returns 1 print(cos(params, params)) # Small deviation from the original --> close to 1, but smaller print(cos(params, params+(torch.rand(params.shape)/100) )) # Some random value --> close to 0 print(cos(params, torch.rand(params.shape)))
Hope this helps