Two set of batches from two different dataloaders

1

How do i get two set of batches at the same time from two different dataloaders for iteration,
for example we can get batches of data from a single data loader as

for i,data in enumerate(dataloader,0):
inputs,labels = data

but i need to get two batches simultaneously from two data loaders, something like
for i,j,data_1,data_2 in enumerate(dataloade1,dataloader2,0):
inputs1,labels1 = data_1
inputs2,labels2 = data_2

2

You can use zip:

datasetA = TensorDataset(torch.randn(10, 1), torch.randn(10, 1))
datasetB = TensorDataset(torch.randn(10, 1), torch.randn(10, 1))
loaderA = DataLoader(datasetA, batch_size=5)
loaderB = DataLoader(datasetA, batch_size=5)

for (dataA, targetA), (dataB, targetB) in zip(loaderA, loaderB):
    print(dataA.shape)

to iterate both DataLoaders.

3

Hai ptrblck,
i found this run time error.
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File “/home/sinnu/.local/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py”, line 302, in _worker_loop
data = fetcher.fetch(index)
File “/home/sinnu/.local/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py”, line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File “/home/sinnu/.local/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py”, line 58, in
data = [self.dataset[idx] for idx in possibly_batched_index]
RuntimeError: CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

4

It seems you are using CUDATensors inside the Dataset, which then fails while trying to initialize multiple contexts. Load the data on the CPU, change the startmethod of multiprocessing to spawn, or use the main thread (num_workers=0).

5

Yeah, it worked, but if you use num_worker = 0 and pin_memory = False , we are blocking the option of making GPU fast. One of the reasons, i use dataloader is to make training fast by making num_worker = 50 and pin_memory = True. Now i dont have these options? then how to solve this problem with these options also. Also can you tell what is startmethod of multiprocessing?

6

You are creating CUDATensors inside the Dataset so setting pin_memory=True would be useless as the data is already on the GPU.
I’ve suggested a few alternatives, one which is to use tensors on the CPU allowing you to use multiple workers and pinned memory.