Will 0 padded entries be updated during training for torch.nn.Embedding?

nlp
1

Hi Team,
Just out of curiosity, for torch.nn.Embedding, if we set padding_idx=0, will the first entry of embedding table be updated during model training(cause its special entry just for alignment and dry-run in models and would be unnecessary if we really want to make all weights of embedding table trainable).

Regards,
Zan

2

No, the weight row will be initialized as all zeros and won’t get any gradients as seen here:

emb = nn.Embedding(10, 5, padding_idx=0)
x = torch.arange(10).view(1, -1)
out = emb(x)
out.mean().backward()
print(emb.weight)
print(emb.weight.grad)
1 Like
3

Thank you Piotr for the answer! PyTorch is really AWESOME :metal:t2: