I have three datasets each loaded using the Dataloader class. The goal is to grab a batch from each data loader, compute the loss on each batch and aggregate all the losses, before moving to the next batch.
What I did was
d1 = []
for batch_idx, samples in enumerate(train_loaders[0]):
if batch_idx >= num_batches:
print("breaking out")
break
d1.append({"images": samples["img"].float(), "targets": samples["lab"]})
With the above I am able to grab any batch. However, this results in memory issues, as the entire data has to be kept in memory.
Is it possible to get a cleaner and memory-efficient solution to this? Thank you for your help. Please @ptrblck any solution?