I am training word embeddings with our new model.
The problem I am facing is my input indexs are in different length.
For example, the input index might be [LongTensor([1,2,3]), LongTensor([5,8])].
Because Embedding class only accept LongTensor, I can not pass in a List.
After some searching, I have two possible solutions:
- Using padding index. However, this method requires a large amount of cuda memory because my index is actually very sparse. For example, the total number of possible index might be around 100000, while each instance only have 100 activated index. So, in this case, padding is not memory efficient.
- Go through the list one by one to index the embedding. This method can work. But this method is too slow. It is at least 10 times slower than batch.
Is there any third method that can resolve my situation?