I would like to train a model on imagenet, but with the default ImageFolder, it is taking too long to train.
To speed up the training process, I want to cache the dataset in RAM. I’ve seen that one way to cache the dataset is to create a large numpy tensor or a list of tensors such as what is done in small datasets (Cifar, Mnist) by torchvision.datasets module.
However, I was wondering how to do that in multiprocessing with distributed data-parallel, because how I understood it is that each process will create an object dataset and I risk duplicating the RAM usage for each process started. Is there a way to cache only once the list of tensors that will then be shared to each process ?