Validation hangs up when using DDP and syncbatchnorm

[sunshichen (Shichen)]

Actually. I met another problem after I upgrade to V1.7.0. The result cames to be much worse than it on 1.1. Could you help me with that?

After Update pytorch from 1.1.0 to 1.7.0. DDP Model cannot converge

I trained a 3D unet model by Pytorch 1.1.0 Distributed Data Parallel. The test result is good. But after I update Pytorch to V1.7.0. Same code different result. Anyone can help me with that? Is it a syncbatchnorm problem or what? Even if I finetune the pretrained model(on 1.1.0). Both losses during training and validation increased a lot using v1.7.0. Below is the training epoch loss of two version. [loss]