Create dataset from a python list of path

[Camille (Petri)]

Good day to all.
I’m currently building a NLP model for document classification.
I have limited amount of computation power and storage, so I was wondering if it was possible to create a pytorch dataset from a python list of path, where every path leads to a different document.
I apologise if my question is not clear enough, and am prepared to give any more information that could be helfpful.
Thank you for your time.
Camille

[Balamurali_M (balamurali)]

Dataloader for semantic segmentation

I am assuming there is a folder which contains sub-folders image and mask. The filenames are same. This is a template you can modify it and use. All the best. import torch.utils.data as data class DataLoaderSegmentation(data.Dataset): def __init__(self, folder_path): super(DataLoaderSegmentation, self).__init__() self.img_files = glob.glob(os.path.join(folder_path,'image','*.png') self.mask_files = [] for img_path in img_files: self.mask_files.a…

Follow this, it is done for semantic segmentation. I think a similar thing can be worked out for your case.