Guidelines for assigning num_workers to DataLoader

73

To me, after some practicality checks, the following worked smoothly:

num_workers attribute in torch.utils.data.DataLoader should be set to 4 * num_GPU, 8 or 16 should generally be good:

  1. num_workers=1, up to 50% GPU, for 1 epoch (106s ovre 1/10 epochs),
    training completes in 43m 24s

  2. num_workers=32, up to 90% GPU, for 1 epoch (42s over 1/14 epochs),
    training completes in 11m 9s

  3. num_workers=8, 16, up to 90% GPU, (8 is slightly better) for 1 epoch (40.5s over 1/16 epochs),
    training completes in 10m 52s

  4. num_workers=0, up to 40% GPU, for 1 epoch (129s over 1/16 epochs),
    training complete in 34m 32s

3 Likes