Embeddings shows index out of range if `padding_id` is -1

nlp
1

Overview

I want to use -1 as an ignore index of an embedding layers.
But it does not work.

Environment

Python: 3.7.1
torch: 1.1.0

To Reproduce

embedding = nn.Embedding(10, 3, padding_idx=-1)
tokens = torch.tensor([-1, 2]).long()
embedding(tokens)

The third line, in the above, shows

RuntimeError: index out of range at ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:193
2 
>>> embedding = nn.Embedding(10, 3, padding_idx=-1)
>>> tokens = torch.tensor([1, 2]).long()   # positive indexes here
>>> embedding(tokens)
tensor([[-0.6424, -1.2020,  0.4287],
        [ 0.5234,  1.2113,  0.8808]], grad_fn=<EmbeddingBackward>)
3

Thank you for your reply.

I think padding_idx=-1 means -1 map to [0., 0., 0.], as the example of https://pytorch.org/docs/stable/nn.html#embedding, but is it not correct?

Probably, does padding_idx=-1 mean there are no padding index?

4

I think -ve padding means negative indexing.

1 Like
5

Just set

tokens = torch.tensor([3, 2]).long()

This may be the embedding index problem <0.
Negative is bad in here.
Also, by default is long so you don’t need to use long()

1 Like