Hi,
I have a very big dataset with a lot of images. Is there any way faster than loading single images in getitem method? I want to increase the batch size and want to find the bottleneck. the batch size, size of my network, …
Thank you
Hi,
I have a very big dataset with a lot of images. Is there any way faster than loading single images in getitem method? I want to increase the batch size and want to find the bottleneck. the batch size, size of my network, …
Thank you
The loading of images onto RAM and GPU can be speed up through custom loaders but speeding the training through larger batch sizes is something different.
if you just want to speed up the load of images, have a look here on copying the data to GPU in parallel.
So using this way I can increase the training speed right? My final goal is to train the network faster. I added a second RTX 2080Ti and use nn.DataParallel and now it’s almost twice faster, but I want it to be faster.
Maybe you can use hdf5
or lmdb
format.
The tools come from NVIDIA---->DALI . But I never use it
yes, it should make the training faster by loading the data faster. You might can do other optimizations to make faster as well, e.g. batch size, network size etc.