Regarding the timing of synchronization of model's weights when trained using DDP

distributed
1

Hello.
I have recently been trying my hand at learning with Data Parallel using Distributed Data Parallel (DDP).

I understand that learning with DDP is done by creating replicas of the model on multiple devices (e.g. GPUs), splitting the data and training them, and synchronizing the weights. My question is, when are the weights synchronized between replicas? At each epoch?

I would appreciate it if you could enlighten me.
Best Regards.

2

DDP will broadcast the state_dict during the construction of the DDP object as described in the DDP - Internal Design docs.

3

Thank you for reply.
I understand about broadcasting.