Multiple dataloaders

1

hello, i am using this command to get the batches from 2 different datat loaders.
Is this correct? Thank you

first method

for iter_id, item1 in enumerate(tqdm(train_loader)):
    iter_id2, item2 = enumerate(target_loader).__next__()
    image, labels = item1
    image2 = item2

    print(image.shape, image.dtype)
    image = np.transpose(image, (0, 2, 3, 1))
    plt.imshow(image[2])
    plt.show()

    print(image2.shape, image2.dtype)
    image2 = np.transpose(image2, (0, 2, 3, 1))
    plt.imshow(image2[2])
    plt.show()

second method

for (item1, item2) in zip(cycle(train_loader), target_loader):
    image, labels = item1
    image2 = item2

    print(image.shape, image.dtype)
    image = np.transpose(image, (0, 2, 3, 1))
    plt.imshow(image[2])
    plt.show()

    print(image2.shape, image2.dtype)
    image2 = np.transpose(image2, (0, 2, 3, 1))
    plt.imshow(image2[2])
    plt.show()
2

This code is confusing. Imagine if you forget to disable shuffle in one of the dataloaders or use multiple workers.

Simpler way here is to define your custom dataset with all the data. Also you can move all preprocessing outside of the loop either with dataset’s init/get_item or with collate_fn.

1 Like