Custom DataLoader unexpected behaviour

data
1

I am trying to combine codes of two repos.
From the first I get the data & its dataloader. From the second the models.

Now in my custom loader I have

train_index, valid_index = train_test_split(range(len(train_data)), test_size=args.validation_split)
        self.train_set = Subset(train_data, train_index)
        self.train_loader = torch.utils.data.DataLoader(
            self.train_set,
            batch_size=batch_size, shuffle=shuffle, batch_sampler=batch_sampler,
        )
        self.valid_dataset   = Subset(train_data, valid_index)
        self.valid_loader = torch.utils.data.DataLoader(
            self.valid_dataset,
            batch_size=batch_size, shuffle=False, batch_sampler=batch_sampler,
        )

        self.test_loader = []
        for id in self.section_id_list:
           _test_loader = DCASE202XT2Loader(
                data_path,
                dataset_name=dataset_name,
                section_keyword=section_keyword,
                machine_type=machine_type,
                train=False,
                section_ids=[id],
                frames=args.frames,
                n_mels=args.n_mels,
                frame_hop_length=args.frame_hop_length,
                n_fft=args.n_fft,
                hop_length=args.hop_length,
                power=args.power,
                fmax=args.fmax,
                fmin=args.fmin,
                win_length=args.win_length,
                data_type=data_type,
                is_auto_download=args.is_auto_download,
           )

           self.test_loader.append(
                torch.utils.data.DataLoader(
                    _test_loader,
                    batch_size=_test_loader.n_vectors_ea_file, shuffle=False
                )
           )

later I get the loaders as


# Get train data loader
train_loader = dataset.train_loader
...
# Get test data loader
test_loader = dataset.test_loader

Now in training I go in a loop and get the input. Works well, my training starts.

for batch_idx, data in enumerate(tqdm(train_loader)):
    inputs = data[0]

In my test I would like to do the same, but I get the “DataLoader not subscritable” error.
Then I assumed, that as I get a DataLoader and my custom get_item has five return values, I need to unpack.

for batch_idx, data in enumerate(tqdm(test_loader)):
    input, label, _, _, idx = data

But this also results in “two many values … expected 2” Error. Thus data should not be unpacked.

I am a bit lost how to get my test loader running.

Should I subscript it when I assign


test_loader = self.test_loader[0] as self.test_loader is actually a list?

Also could I go for this approach in my trainloader? Would look a bit more sexy

for batch_idx, data in enumerate(tqdm(train_loader)):
    inputs, _, _, _, _ = data
2

You should iterate a DataLoader instance instead of a list containing DataLoaders unless you want to use a nested loop and get the data of each test_loader.

dataset = TensorDataset(torch.randn(100, 1), torch.randn(100, 1))
loader = DataLoader(dataset, batch_size=50)

for idx, (data, target) in enumerate(loader):
    print(data.shape)
# torch.Size([50, 1])
# torch.Size([50, 1])

test_loader = []
for _ in range(3):
    test_loader.append(DataLoader(dataset, batch_size=50))

for i, loader in enumerate(tqdm(test_loader)):
    for idx, (data, target) in enumerate(loader):
        print(data.shape)
# torch.Size([50, 1])
# torch.Size([50, 1])
# torch.Size([50, 1])
# torch.Size([50, 1])
# torch.Size([50, 1])