Will it be faster if I convert the jpg datasets into pt datasets?

1

I am going to train ImageNet on a single GPU, but it is too large to put all of them in the memory for CPU. So, I’m just thinking what if I convert all the jpg files into transforms.Resized pt datasets, as I don’t need to go through PIL.Image.open and transforms.Resize. Will this be faster? Or there’re some quicker methods?
Owing to the large size of the dataset, I don’t have much time to test all methods that I can imagine. Therefore, any help is appreciated. :innocent:

2

You wouldn’t have to profile the entire dataset, but just a subset estimating the expected performance. E.g. while storing tensors could save some processing in e.g. resizing you should consider if you want to use static resizing during training or a random crop followed by resizing. Also, the size difference might also be large comparing JPEG encoded images against raw tensors as described here.

3

Thanks for your suggestions!