About Synchronize Batch Norm across Multi-GPU Implementation

1

I want to implement synchronize batch norm across multi-GPU. How can I do it? I think I should synchronize its mean and variance both forward and backward pass, so can I use the register_hook ? Can someone give me some advise? Thank you.

2 Likes
2

I’m not sure on the implementation, but for what it’s worth it’s actually recommended NOT to try and synchronize batchnorm for multi-node training. Not to deter you from exploring it, but unless you’re specifically investigating it I’d say it’s not the best place to devote effort for performance improvements.

3

Hmmm that’s weird. Isn’t that the exact opposite of DeepLab v3 and PSPNet, where their secret-sauce is essentially the fine-tuning of BN parameters on the VOC dataset across multiple GPUs?

2 Likes
4

Yes, the PSPNet use this BN. I am re-implementing the network and train process. So I have to implement this BN.

5

A brief description of implementing synchronize BN:

5 Likes
6

Thank you. I just think about how to communicate across GPUs.

7

Basically, you need to customize the DataParallel and enable the cross GPU communication for each layer. Good luck.

2 Likes
8

Just to update, I have released the Synchronized Mulit-GPU Batch Normalization http://hangzh.com/PyTorch-Encoding/syncbn.html

9

Any updates on a Synchronized Batch Norm in Pytorch?

ChainerMN has implemented one here

10

Please checkout the PyTorch compatible Synchronized Cross-GPU encoding.nn.BatchNorm2d and the example.

1 Like