I have trained my model with DDP, use sync batchnorm for multiple times. The model accuracy is consistently 0.5% lower than the model trained with single gpu. Is it true to say that DDP will degrade the performance of the model?
I have trained my model with DDP, use sync batchnorm for multiple times. The model accuracy is consistently 0.5% lower than the model trained with single gpu. Is it true to say that DDP will degrade the performance of the model?