Custom dataset with multiple datasets

vision
1

Hi
I have multiple datasets, I want to sample them with some weights like 2x dataset1 1x dataset2, … and then define a dataloader for this concatenated dataset. How I can define a custom dataset which can have sampling mechanism?
Also the datasets are very large, and I cannot bring them to memory. thanks

After making concateneted sampled dataset, I want to define 1 dataloader for it, also I need to make sure each batch comes from one type of dataset. thanks

2

You could use a WeightedRandomSampler for the concatenated datasets.
If you need to sample a complete batch from one of the internal datasets only, you could create a custom sampler, create the indices internally and return the batch indices to the custom dataset.