# DDP - Batch Norm Issue

**URL:** <https://discuss.pytorch.org/t/ddp-batch-norm-issue/80751>\
**Category:** distributed\
**Created:** [May 11, 2020, 8:12pm UTC](https://discuss.pytorch.org/t/ddp-batch-norm-issue/80751 "2020-05-11T20:12:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![soulslicer](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soulslicer/32/9454_2.png) [@soulslicer](https://discuss.pytorch.org/u/soulslicer)\
**Post date:** [May 11, 2020, 8:12pm UTC](https://discuss.pytorch.org/t/ddp-batch-norm-issue/80751/1 "2020-05-11T20:12:59Z")

</div>

I am having the issue that everyone else has, where a model that uses BatchNorm has poorer accuracy when using DDP:

According to this, I am suppose to patch Batch Norm somehow:

> <https://github.com/Microsoft/human-pose-estimation.pytorch/issues/8>
>
> Thank you for releasing the code. The README reads that it is required to disable cudnn for batch\_norm. Would you please...

```auto
def monkey_patch_bn():
    # print(inspect.getsource(torch.nn.functional.batch_norm))
    def batch_norm(input, running_mean, running_var, weight=None, bias=None,
                   training=False, momentum=0.1, eps=1e-5):
        if training:
            size = input.size()
            size_prods = size[0]
            for i in range(len(size) - 2):
                size_prods *= size[i + 2]
            if size_prods == 1:
                raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))

        return torch.batch_norm(
            input, weight, bias, running_mean, running_var,
            training, momentum, eps, False
        )
    torch.nn.functional.batch_norm = batch_norm

```

But I am not sure how to do it if my code is like this:

```auto
def convbn(in_planes, out_planes, kernel_size, stride, pad, dilation, bn_running_avg=False):
    return nn.Sequential(nn.Conv2d(in_planes, out_planes,
                                   kernel_size=kernel_size, stride=stride,
                                   padding=dilation if dilation > 1 else pad,
                                   dilation=dilation, bias=False),
                         nn.BatchNorm2d(out_planes,
                                        track_running_stats=bn_running_avg))

```

---

<div class="post-metadata">

**Author:** ![rvarm1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rvarm1/32/15821_2.png) [@rvarm1](https://discuss.pytorch.org/u/rvarm1)\
**Post date:** [May 11, 2020, 10:59pm UTC](https://discuss.pytorch.org/t/ddp-batch-norm-issue/80751/2 "2020-05-11T22:59:30Z")

</div>

Hi,

Could you try disabling the CuDNN backend with:  
`torch.backends.cudnn.enabled = False`? According to posts such as [Training performance degrades with DistributedDataParallel](https://discuss.pytorch.org/t/training-performance-degrades-with-distributeddataparallel/47152/13), can improve training.

Also, have you given `SyncBatchNorm` ([https://pytorch.org/docs/stable/nn.html#syncbatchnorm](https://pytorch.org/docs/stable/nn.html#syncbatchnorm)) a try? This will make batch statistics be computed across all GPUs in usage, instead of being computed separately for the batches passed to each device. (Note that as per the documentation, you’ll have to change your code to spawn a single process per-GPU if you’re not training that way already)

---

<div class="post-metadata">

**Author:** ![soulslicer](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soulslicer/32/9454_2.png) [@soulslicer](https://discuss.pytorch.org/u/soulslicer)\
**Post date:** [May 12, 2020, 4:13pm UTC](https://discuss.pytorch.org/t/ddp-batch-norm-issue/80751/3 "2020-05-12T16:13:42Z")

</div>

Okay but this is only useful if i am having running\_mean\_stats enabled right?