Hello guys, I am new to Pytorch. I am trying to load a large dataset of images(around 5 million images). I use the DataLoader to load the images. But it takes so much time to load and process 1 epoch (1 epoch roughly takes 10 hours). I use 2 V100 GPUs for this process. I went through some answers but could not get them. Kindly let me know if there is any way to do this efficiently. Thank you
You could profile your code and try to isolate, if the data loading is the bottleneck.
If that’s the case, this post might be helpful to further optimizer the loading.
There’s a few ways to optimize the training time but if you already implemented them there isn’t much you can do except get better GPUs.
Have you tried:
- resizing you images
- cropping your images
- augmenting the batch size
- augmenting the number of workers
- freezing the model, only using the last layer
Personally, this previously got me from 3hrs per epoch to 10min.