Generally, multiple workers should speedup your data loading, although the sweet spot depends on your setup.
You could have a look at this post for more information about potential bottlenecks.
Also, you might want to use DALI in case it fits your use case.