The one-hot encoded input tensors represent a sequence of pos tags.
One input line is composed by (for my simplest model) Three distance numbers, and 6 pos tags which are encoded as one-hot vectors. It gives me a ~195 tensor which is composed by mostly zeros.
In this condition, do you think it’s a good idea to use a nn.Embedding layer at the beginning of my network ?