Hi everyone, I am seeking help on how to effectively write a data loader for ImageNet. I store the ImageNet-1K dataset (with *.zip) format on my local disk. However, I cannot unzip it to my local disk due to the limitation of the number of files. I wrote a data loader that can get items from *.zip file, but it is too slow. Also, that data loader doesn’t allow me to do any distributional training.
The GPU node I can use has a big RAM (1TB), I am wondering whether could I unzip all the training data to the RAM, and then let the dataloader directly get data from RAM?
P.S.: Someone told me that I could mount a memory disk before training, but I do not have the right to do this.
Thanks very much.