Hello everyone, - currently I’m training EfficientNet-B0-ns (it’s kinda new SOTA ResNet) model for some classification task, num_classes = 5.
My GPU, which is RTX 3060Ti allows me to choose batch size up to 64 in order to completely saturate GPU VRAM.
Also I have 6 core Ryzen 5 2600x, - and even with num_workers set to 12 it can’t saturate the GPU (I can see how the model is struggling to compute something, because it’s not ready yet, via tqdm).
Is there any way we can prevent CPU bottlenecking the GPU in case of small models?
Assuming the data loading is the bottleneck, you could take a look at this post, which explains common use cases and potential workarounds.
If not already done, use multiple workers in the DataLoader.
Thanks for the useful post, but the actual problem was my code.
I was applying huge augmentations before resizing the image, not vice versa, - resolution of image on which augmentations was performed was like 800x600 instead of 224x224, and that caused the bottleneck.
Changing the order of augmentations and resizing fixed my problem, for sure
