Standard ways to initialize weights?

eg if I have embedding = nn.Embedding(V, E), is the shortest method something like:

scale = my_hand_written_formula_here, scale)

(and ditto for bias too, eg in a Linear).


I think you are missing the underscore to call uniform inplace. Besides that, that would be one way.
I would recommend to avoid using .data and use something like this instead:

with torch.no_grad():
    embedding.weight.uniform_(-scale, scale)

Ok. How to get scale? Do I need to hunt down an appropriate formula? (I know a couple, just wondering if there is an easy way that avoids having to do this?).

If none of the implemented init functions provides what you need, you would have to implement it somehow manually.
However, there are already a lot of init functions like xavier_uniform etc.
Also, maybe it’ll help to have a look at the code to get some idea how to calculate the scale etc.

Ok. missing sqrt(3) / sqrt(num_inputs) * factor initialization, eg see

(But does otherwise seem quite comprehensive :slight_smile: And maybe I just overlooked this one in the list somewhere, eg not written quite in the same formula, but is equivalent).