How do I train using multiple data loaders

vision
1

Hi all.

I am trying to train my model using various image sizes and batch sizes.

I want to train my model using multiple dataloaders.

Here’s how I can simply think about it in code.

dataloader1(image_size=(1024, 512), batch_size=2,...)
dataloader2(image_size=(512, 256), batch_size=4,...)
dataloader3(image_size=(256, 128), batch_size=8,...)

for epoch in range(0, epochs):
    for idx, (input, target) in enumerate(dataloader1):
            loss1
    for idx, (input, target) in enumerate(dataloader2):
            loss2
    for idx, (input, target) in enumerate(dataloader3):
            loss3

    tot_loss = loss1 + loss2 + loss3
    tot_loss.backward()

The method that comes to mind right now is the same as above. Does the above method work properly? Or is there another efficient way?

In summary, I want to train one model using multiple data loaders, add up each loss and update it back to one model.

Please let me know if there is any possible example or method.

Thank you.

1 Like
2

Your posted code could work assuming your model is small and you have enough memory.
Assuming you are summing the losses inside the DataLoader loop, this approach would store the computation graphs, which would allocate more memory in each iteration.
Depending on the size of your dataset, you might be quickly running out of memory.
As an alternative, you could call backward() on each loss in each iteration, so that Autograd can free the intermediate tensors, as they are not needed anymore.

1 Like
3

Thank you for answer.

I have additional questions. If you want to use memory efficiently when trying to learn using multiple dataloaders, is there any way to modify the inside of the dataloader?

Or Is there a way to share the queue of each data loader?

4

You could concatenate the datasets via ConcatDataset and pass it to a single DataLoader.
However, you would have to make sure that the output sizes of all tensors are equal (doesn’t seem to be the case in your example) or you would have to use a custom collate_fn.

1 Like
5

Thanks for the good answer.

But what I want to do is use multiple data loaders. Sorry, is there any way to utilize each data loader’s queue??

6

Each DataLoader would use its own queue and I don’t think you can share it between different loaders.

1 Like
7

I’ll ask you one more thing.

If so, is there any other way to receive and train different image sizes and different batch sizes as inputs except collate_fn, concatdataset??

As far as I know, the batch size should be the same when concatdataset is applied.

Thank you

1 Like
8

If you are using different loaders, as posted in your initial code, then yes, each loader can use different input shapes and batch sizes.
If you concatenate the dataset and use a single loader, the batch size would be set in the DataLoader creation unless you disable automatic batching and customize the loading pipeline.

1 Like
9

Sorry.

I don’t understand the second paragraph a bit. Could you please tell me more specifically?

Thank you