Differences between nn.Embedding and nn.Parameters?

vision
1

Hi there,

What’s the differences between nn.Embedding and nn.Parameters when I only need to access the the word embedding by indices?

Suppose:

  1. the word indices are W = [1, 2, 3, 4, 5, 0], which is a LongTensor.
  2. E = nn.Embedding(10, 2)
  3. P = torch.Tensor(10, 2).uniform_(-0.1, 0.1).requires_grad_()

If I want to access the embedding of the corresponding words. what’s the differences between the following two ways:

E(W)

and

P[W, :]

?

Any efficiency differences?

1 Like
2

I also would like to know the difference

3

Here is a nice explanation:

they are actually all the same underneath, just a trainable matrix (linear comes with an extra bias tensor). however, they have wrappers that allow them to behave differently when you give it an input.

  1. nn.Embedding selects the rows of the given matrix, given a list of integers
  2. nn.Linear does the einsum operation ...d, d e -> ...e
  3. nn.Parameter basically just makes a tensor trainable (receive gradients and updates on step). this is the lowest level you can go, so actually, you can define your entire deep neural network with just nn.Parameters and manually do all the above with gathers and matrix multiplies or einsums
1 Like