It depends, where the actual bottleneck is.
You could use the ImageNet example to get the actual data loading time. If it stays at a high value, you might have a data loading bottleneck.
If that’s the case, have a look at this post to see some potential workarounds.
On the other hand, if you see the data loading time approaching zero, your model might create the bottleneck, in which case you could try to profile it (e.g. using nsight) and see, which operations are the slowest.
The number of workers should speed up the data loading time. However, there is usually a sweet spot, after which increasing the number of workers might slow down the code again.