# DistributedDataParallel and convert\_sync\_batchnorm

**URL:** <https://discuss.pytorch.org/t/distributeddataparallel-and-convert-sync-batchnorm/161070>\
**Category:** distributed\
**Created:** [September 9, 2022, 9:43am UTC](https://discuss.pytorch.org/t/distributeddataparallel-and-convert-sync-batchnorm/161070 "2022-09-09T09:43:49Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![zzzf](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zzzf/32/14158_2.png) [@zzzf](https://discuss.pytorch.org/u/zzzf)\
**Post date:** [September 9, 2022, 9:43am UTC](https://discuss.pytorch.org/t/distributeddataparallel-and-convert-sync-batchnorm/161070/1 "2022-09-09T09:43:49Z")

</div>

I find most people first convert `BatchNorm` to `SyncBatchNorm` and then wrap the `model` with `DistributedDataParallel`:

```auto
model = nn.SyncBatchNorm.convert_sync_batchnorm(model)
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank])        

```

If I reverse the order like the following, would I get the same results?

```auto
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank]) 
model = nn.SyncBatchNorm.convert_sync_batchnorm(model)

```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [September 9, 2022, 5:35pm UTC](https://discuss.pytorch.org/t/distributeddataparallel-and-convert-sync-batchnorm/161070/2 "2022-09-09T17:35:52Z")

</div>

I would generally stick to the docs and their usage, but based on [this code](https://github.com/pytorch/pytorch/blob/b136f3f310aa01a8b3c1e63dc0bfda8fd2234b06/torch/nn/modules/batchnorm.py#L820-L823) I guess you might see the same result assuming the recursive `named_children()` call also sees the internal `.module` attribute.