Hi, My problem is related to this Train simultaneously on two datasets.
I have two datasets, MNIST(60000) and USPS(9298).
I used torch.utils.data.ConcatDataset ( datasets ) to merge these two.
Now, when I use a batch size of anything other than 1 or 2, after the floor(len(first_dataset)/batch_size)iterations, I get this error,
" invalid argument 0: Tensors must have same number of dimensions: got 1 and 2 at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/TH/generic/THTensorMath.c:3577".
I guess this is because of the torch.stack in default_collate_fn.
How do I write a new_collate_fn to overcome this problem?
I’m not familiar with the USPS dataset, but are the images in there exactly the same size as the mnist dataset?
Also can you use each dataset independently? Do they work properly?
Why do you want to implement in your cistom collate function?
Hi Alban
Yes , in both MNIST and USPS dataset-the size of each sample is [1,28,28].
I am able to use each dataset independently with any batch size I want.
It throws me an error when I used torch.utils.data.ConcatDataset ( datasets ) for anyother batch sizes other than 1 or 2.
I will give an example.
Lets say I have batch size-128. In case of using MNIST alone, then the last iteration shall be of size mod(60000,128)= [96,1,28,28].
I guess this uneven balance is throwing me off when I concatenate the datasets.
File “/media/kowshik/Data1/gan_expts/autoencoder/ae_mnisyt+usps.py”, line 129, in
for count,data in enumerate(mnist_usps_dataloader):
File “/home/kowshik/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 286, in next
return self._process_next_batch(batch)
File “/home/kowshik/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 307, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
File “/home/kowshik/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 57, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File “/home/kowshik/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 138, in default_collate
return [default_collate(samples) for samples in transposed]
File “/home/kowshik/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 138, in
return [default_collate(samples) for samples in transposed]
File “/home/kowshik/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 115, in default_collate
return torch.stack(batch, 0, out=out)
RuntimeError: invalid argument 0: Tensors must have same number of dimensions: got 1 and 2 at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/TH/generic/THTensorMath.c:3577