I am creating my own custom image dataset using torchs Dataset class.
So far, I iterate through all .jpg files in a given folder and store them as a list by appending.
This costs a lot of working memory + it takes ages to load the dataset.
I was wondering what a smart way is to load the images? What is considered good practice when working with a lot of images?
I was thinking about storing the paths where the images are (as .csv / .json) and only load the images in the def getitem(self, idx): method, when they are actually needed.
What’s the most efficient way?
Thanks for any suggestions!