That would be the ideal linear scaling you could achieve, thus reducing the epoch time by number of GPUs.
Too many CPU workers might slow down the data loading. I’m not an expert on this topic, but always refer to @rwightman’s post.
2 Likes