Checkpointing DDP.module instead of DDP itself

[mrshenli (Shen Li)]

@mazabou3

When checkpointing, is it ok to save ddp_model.module instead of ddp_model? I need to be able to use the checkpoint for 1. evaluation using a single gpu 2. resume training with multiple gpus

Yep, this is actually the recommended way:

1. On save, use one rank to save ddp_model.module to checkpoint.
2. On load, first use the checkpoint to load a local model, and then wrap the local model instance with DDP on all processes (i.e., sth like DistributedDataParallel(local_model, device_ids=[rank])).