Validation hangs up when using DDP and syncbatchnorm

[pritamdamania87]

@sunshichen

Yes. Validate on all GPUs works. But shouldn’t model.eval() aotomatically disable sync process?

Good point, I’ve opened an issue for this: SyncBatchNorm should avoid sync when model.eval() is True. · Issue #48988 · pytorch/pytorch · GitHub