Custom weight intialization

1

I am sorry if this is a very simple question. How do I initialize the weights of my network such that the weights are produced by the softmax function on the learnable log-probabilities?

2

You could create the desired initial weights using any method and copy them into the model parameter via:

my_weight = ... # your method to create the initial weights
with torch.no_grad():
    model.my_layer.weight.copy_(my_weight)
1 Like