I have saved 80000 PyTorch tensors of shape (64, 64) in some folder in pt format. because of the memory limitation, I cant load all of these files at once so I want to use this folder as the root and load the data in batches like what is done with images dataset using torchvision.datasets.ImageFolder, Is there some similar PyTorch function that does this for me?
If the structure of your data is equal to what ImageFolder expects (i.e. samples for classes are located in their corresponding folder), you could use torchvision.datasets.DatasetFolder and define a custom loader method, which could be as simple as lambda path: torch.load(path). You can check the default loaders here.
If that’s not the case and your data is located in a single folder, you could write a custom Dataset instead as described here.