Dataloader for different datasets

Hi, I’m trying to finetune a model using custom datasets, however, the datasets contain different annotation styles. One dataset contains its annotations in a JSON File, another has the data saved in the pixel value of the image, and another contains them in H5 files. What is the easiest way for me to combine all of these into one dataset so that I can properly fine tune my model? Or is it best to create 3 different data loaders, and then randomly chose which one to use for the next training image?

I assume that the output of the Dataset.__getitem__ would still yield the same objects regardless how the data is loaded internally (from the JSON file, H5, or directly as images).
If that’s the case you could write different Dataset implementations and use ConcatDatset to create a single one before passing them to the DataLoader.

1 Like

Ah, okay great. Thanks a lot, I didn’t realize that you could write a different dataset class and concat them for a data loader. Must have missed it in the docs. :slight_smile: