Should we split batch_size according to ngpu_per_node when DistributedDataparallel

[junb]

@ekurtic

However, multiplying after should be fine too but you have to be careful and multiply before the optimizer.step() is called. I guess this second approach can then be achieved in two different ways: either by multiplying gradients (which could be slow) or by multiplying your learning_rate so that the final weight update has a proper scale.

Good call, I will have to remember that! Thanks for your quick answer.