Could you check, if the data loading pipeline is a bottleneck and thus your GPU could be starving?
Just execute the training loop with predefined random data on the GPU (don’t use your real data) and check the GPU utilization.
1 Like