# \[DPP\] define optimizer before DPP or after DPP (plus amp)

**URL:** <https://discuss.pytorch.org/t/dpp-define-optimizer-before-dpp-or-after-dpp-plus-amp/114323>\
**Category:** distributed\
**Created:** [March 10, 2021, 9:52am UTC](https://discuss.pytorch.org/t/dpp-define-optimizer-before-dpp-or-after-dpp-plus-amp/114323 "2021-03-10T09:52:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![111429](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/1/e0b2c6/32.png) [@111429](https://discuss.pytorch.org/u/111429)\
**Post date:** [March 10, 2021, 9:52am UTC](https://discuss.pytorch.org/t/dpp-define-optimizer-before-dpp-or-after-dpp-plus-amp/114323/1 "2021-03-10T09:52:54Z")

</div>

Hi, a question about the appropriate ordering of defining optimizer in DPP + amp scenario.  
Define optimizer before DPP ( `torch.nn.parallel.DistributedDataParallel`) :

```auto
optimizer = torch.optim.AdamW(model.parameters(), lr=0.01)
model, optimizer = amp.initialize(model, optimizer, opt_level='O2')
model = DDP(model)

```

define optimizer after DPP :

```auto
model = DDP(model)
optimizer = torch.optim.AdamW(model.parameters(), lr=0.01)
model, optimizer = amp.initialize(model, optimizer, opt_level='O2')

```

The official video classification [script](https://github.com/pytorch/vision/blob/master/references/video_classification/train.py#L195) adopts the first logic.  
Yet I see some blog says that you should initialize an optimizer over a DPP model when you use DPP (though no amp context involved in that article).  
I wonder which one is correct. or dosen’t matter, both are fine. Thanks.

---

<div class="post-metadata">

**Author:** ![rvarm1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rvarm1/32/15821_2.png) [@rvarm1](https://discuss.pytorch.org/u/rvarm1)\
**Post date:** [March 11, 2021, 12:20am UTC](https://discuss.pytorch.org/t/dpp-define-optimizer-before-dpp-or-after-dpp-plus-amp/114323/2 "2021-03-11T00:20:29Z")

</div>

DDP does not change `model.parameters()`, and the optimizer works entirely locally, so defining the optimizer before or after wrapping model with DDP should not make a difference.