Torchtext not processing my data

I am using this custom torchtext data loader .

def load_dataset_file(filename):
    with, "rb") as f:
        loaded_object = pickle.load(f)
        return loaded_object

class SignTranslationDataset(data.Dataset):
    """Defines a dataset for machine translation."""

    def sort_key(ex):
        return data.interleave_keys(len(ex.sgn), len(ex.txt))

    def __init__(
        path: str,
        fields: Tuple[RawField, RawField, Field, Field, Field],
        """Create a SignTranslationDataset given paths and fields.

            path: Common prefix of paths to the data files for both languages.
            exts: A tuple containing the extension to path for each language.
            fields: A tuple containing the fields that will be used for data
                in each language.
            Remaining keyword arguments: Passed to the constructor of
        if not isinstance(fields[0], (tuple, list)):
            #print("ISL uses first")
            fields = [
                ("sequence", fields[0]),
                ("signer", fields[1]),
                ("sgn", fields[2]),
                ("gls", fields[3]),
                ("txt", fields[4]),


        if not isinstance(path, list):
            path = [path]

        samples = {}
        for annotation_file in path:
            tmp = load_dataset_file(Path(annotation_file).expanduser())
            for s in tmp:      
                seq_id = s["name"]
                if seq_id in samples:
                    assert samples[seq_id]["name"] == s["name"]
                    assert samples[seq_id]["signer"] == s["signer"]
                    assert samples[seq_id]["gloss"] == s["gloss"]
                    assert samples[seq_id]["text"] == s["text"]
                    samples[seq_id]["sign"] =
                        [samples[seq_id]["sign"], s["sign"]], axis=1
                    samples[seq_id] = {
                        "name": s["name"],
                        "signer": s["signer"],
                        "gloss": s["gloss"],
                        "text": s["text"],
                        "sign": s["sign"],

        examples = []
        for s in samples:
            sample = samples[s]
                        # This is for numerical stability
                        sample["sign"] + 1e-8,

        super().__init__(examples, fields, **kwargs)

A sample of my dataset file is given below:

[{'name': '0', 'signer': 'Signer0', 'gloss': 'are you free today', 'text': 'are you free today', 'sign': tensor([[[[0.2514]],

























However when I check the len of train data it outputs zero, but when I check the len of my val and test data they are not empty. I tried to make the val data as my train but the len turned zero. Any ideas for me or things I should check?

Thank you in advance

I’ve got an update when I don’t filter by max_sent_length, I can get the correct len(data):

    train_data = SignTranslationDataset(
        fields=(sequence_field, signer_field, sgn_field, gls_field, txt_field),
        #filter_pred=lambda x: len(vars(x)["sgn"]) <= max_sent_length
        #and len(vars(x)["txt"]) <= max_sent_length

However, I get the following error:

  return torch.stack([torch.stack(ft, dim=0) for ft in features], dim=0)
RuntimeError: stack expects each tensor to be equal size, but got [1024, 7] at entry 0 and [1024, 3] at entry 1

Any ideas for me? Thank you in advance