I have a few TB of tiny images and I would like to scan/classify them with a model I have trained. Currently, I load a bunch to memory, create a DataLoader object, run them through the model, and move to the next bunch. By using the DataLoader I get to use the same image transformation I used for the model. However, this is a little painful. Is there a better way of running a model over millions of files?
You could use a Dataset to lazily load all images.
Have a look at this tutorial.
Basically you can pass the image paths to your Dataset and just load and transform the samples in __getitem__.
That was easy, I got it to work. I keep forgetting how flexible PyTorch is compared to other frameworks. Thanks @ptrblck.
I also have a large data set containing series, meaning that each sample of the data set is a sequence, and the data set does not fit into memory. I am looking for a way to handle this efficiently.
@ptrblck if I understand your answer correctly, you suggest that each sample of the dataset will be a separate file, so that it has its own path, which will be given to the DataLoader. Doing that, each sample will be read each time (regarding epochs) from the disk. This would be very efficient considering RAM, but will have poor running time performance.
Wouldn’t be better if the dataset was split into chunks that fit into memory, and iteratively loading these chunks to memory, and then the DataLoader loading them from RAM?
This is the first time that I face RAM problem, so it would help a lot if you have any best practices regarding large datasets that don’t fit into memory.