Will 0 padded entries be updated during training for torch.nn.Embedding?

Hi Team,
Just out of curiosity, for torch.nn.Embedding, if we set padding_idx=0, will the first entry of embedding table be updated during model training(cause its special entry just for alignment and dry-run in models and would be unnecessary if we really want to make all weights of embedding table trainable).


No, the weight row will be initialized as all zeros and won’t get any gradients as seen here:

emb = nn.Embedding(10, 5, padding_idx=0)
x = torch.arange(10).view(1, -1)
out = emb(x)
1 Like

Thank you Piotr for the answer! PyTorch is really AWESOME :metal:t2: