i want to add or take the mean of the embedded vectors after this

self.embeddings = nn.Embedding(vocab_size, embedding_dim)

OR add a linear transformation to every embedded vector (with shared weights of course).

For the first question:

```
embs = self.embeddings(input)
mean_embs = embs.mean(1).squeeze(1)
```

For the second question:

```
embs = self.embeddings(input)
trans_embs = self.linear(embs.view(-1, embedding_dim)).view(embs.size(0), embs.size(1), -1)
```

1 Like