I was using torchtext for creating a text dataset based on a nested field with include length set to true
FIELD= data.Field(lower=True, init_token=sos_token,eos_token=eos_token, tokenize=tokenize, batch_first=batch_first,fix_length=max_length)
NESTED_FIELD=data.NestedField(FIELD,include_lengths=include_lengths)
which produces for example the following structure:
(torch.Size([2, 3, 10]),torch.Size([2]),torch.Size([2, 3]))
where the first entry in triplet contains the data. The second entry contains how many sequences are present in each batch and the third entry contains the information of how long the sequences are.
Consider the following examples with numbers:
tensor([[[ 2, 4, 5, 16, 3, 1, 1, 1, 1, 1],
[ 2, 4, 5, 14, 3, 1, 1, 1, 1, 1],
[ 2, 10, 3, 1, 1, 1, 1, 1, 1, 1]],
[[ 2, 10, 3, 1, 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]])
tensor([3, 1]) --> the first batch contains 3 sequences and the second batch contains only one sequence (1 is ignored)
tensor([[5, 5, 3], -->the 3 sequences of the first batch have the sequence length 5,5 and 3
[3, 0, 0]]) --> the one sequence in batch two has a sequence length of 3
I want to change the structure so that I have the following format:
torch.Size([4,10]) --> only the sequences that carry meaningful data (which are in the above case 4)
torch.Size([4]) --> the length of each of this 4 sequences