Simple example on how BucketIterator works

nlp
1

I have a numerical sequence data, not in csv format. I was wondering if someone could guide me how to make BucketIterator work as I tried many different versions and I’m hopeless now. The documentation is not to helpful on that :confused:

import torchtext
from torchtext.data import Field
from torchtext.data import BucketIterator

SEQ = Field(sequential=True, use_vocab=False, pad_token=99)
LABEL = Field(sequential=False, use_vocab=False)

data = [dict(seq=[0,1,2,1,3,2], lab=(1,0,1)),
        dict(seq=[0,1,0],lab=(0,0,1)),
        dict(seq=[2], lab=(0,0,1)),
        dict(seq=[3], lab=(1,0,1))]

dt = torchtext.data.Dataset(data, fields = {'seq':SEQ, 'lab':LABEL})
bi = BucketIterator(dt, batch_size=2, sort_key=lambda x: len(x['seq']))

Expected behaviour: bi would contain batches of examples grouped by length of sequence and padded with 99 (if neccessary). There should be 2 batches of 2 elements each then.

Observed behaviour: bi returns 4 elements each being an example from my dataset, or errors out saying: AttributeError: 'dict' object has no attribute 'seq'