Let WordEmbedding accept feature vectors

nlp
1

Hello,

as far as I know is the WordEmbedding layer used to store/retrieve embeddings of words for a given indices (depending or embedding size and dict size).

So for example a word is represented as an index and a sentence as a list of indexes.

What if I don’t want to pass indices, but for example a one-hot encoding of my given word(s). So more or less to the encoding manually? I looked at the WordEmbedding class but couldn’t really figure out if subclassing WordEmbedding myself or if I should build my own embedding class (one-hot encoding + linear layer).

I hope it is clear, what I want to achieve.

Greetings,
Patrick

2

Hello Patrick,

you mean the Embedding layer?

If you have one-hot encoded words, you can use torch.nn.functional.linear (which is matrix multiplication, what happens in a a Linear layer without bias) to get the same result, here with

e = torch.nn.Embedding(2, 5)

idx = torch.tensor([[0, 1]]) # batch x seq
print(e(idx)) # batch x seq x embedding

one_hot = torch.nn.functional.one_hot(idx, num_classes=2).float()

print(torch.nn.functional.linear(one_hot, e.weight.t()))

Best regards

Thomas

3

Hello Thomas,

great idea. Thank you for the solution!

Patrick