Compare layers' weights

vision
1

Hi everyone!
I’m a newbie to pytorch and I want to find a pretty way to compare layers’ weights between two models.
I’m currently using an alexnet and I’m freezing all the layers exept for the convolutional layers.

net = alexnet(pretrained=True) 
#freeze all the layers
for param in net.parameters():
    param.requires_grad = False

# unfreeze only the fully connected layers
net.classifier[6] = nn.Linear(4096, NUM_CLASSES) 
for val in net.classifier:
  val.requires_grad = True

After the training phase, I save the best net (the one with the highest accuracy on the validation set) and creates some histograms with tensorboard.

basic_net = alexnet(pretrained=True)
tb = SummaryWriter(comment = "Compare Conv Layers")
j = 0
for i in [0,3,6,8,10]:
  tb.add_histogram("Pretrained conv layer", basic_net.features[i].weight.data.cpu().numpy(), j)
  j = j+1
tb.close

tb = SummaryWriter(comment = "Compare Conv Layers")
j = 0
for i in [0,3,6,8,10]:
  tb.add_histogram("My conv layer", best_net_3_2_1.features[i].weight.data.cpu().numpy(), j)
  j = j+1
tb.close

tb.flush()

With the index i I consider just the convolutional layers.

Annotazione 2020-05-01 153248
Annotazione 2020-05-01 153248313×568 23.4 KB

This is what I obtained in tensorboard, is there any better way to compare the weights?

2

I don’t think displaying two histograms on the same figure is supported with tf-board, but you can have a third hist to show the difference between the two

1 Like
3

Another solution is try WandB (free for personal use) instead of Tensorboard, I recently found it really handful

1 Like
4

Thanks for all the advices, I’m currently only displaying the difference but they are all zeros.

basic_net = alexnet(pretrained=True)
tb = SummaryWriter(comment = "Diff")
j = 0
for i in [0,3,6,8,10]:
  tb.add_histogram("Diff", basic_net.features[i].weight.data.cpu().numpy() - best_net_3_2_2.features[i].weight.data.cpu().numpy(), j)
  j = j+1
tb.close()

tb = SummaryWriter(comment = "Diffb")
j = 0
for i in [0,3,6,8,10]:
  tb.add_histogram("Diffb", basic_net.features[i].bias.data.cpu().numpy() - best_net_3_2_2.features[i].bias.data.cpu().numpy(), j)
  j = j+1
tb.close()
tb.flush()

Is that possible? Am I missing something?