I have saved 80000 PyTorch tensors of shape (64, 64) in some folder in pt format. because of the memory limitation, I cant load all of these files at once so I want to use this folder as the root and load the data in batches like what is done with images dataset using torchvision.datasets.ImageFolder, Is there some similar PyTorch function that does this for me?
If the structure of your data is equal to what
ImageFolder expects (i.e. samples for classes are located in their corresponding folder), you could use
torchvision.datasets.DatasetFolder and define a custom
loader method, which could be as simple as
lambda path: torch.load(path). You can check the default loaders here.
If that’s not the case and your data is located in a single folder, you could write a custom
Dataset instead as described here.