How to transfer single GPU resume to dataparallel mode multi-GPU in train stage with Pytorch

[seungjun]

You can refer to the example code in this reply.

Save checkpoints trained on multi GPUs for load on single GPU

Hi @Janine, This is not related to PyTorch version but the DataParallel (also DistributedDataParallel) class wrapper of PyTorch nn class models. DataParallel encloses the original model as it member variable, self.module. In case you need both single-GPU and multi-GPU model training, you can change saving/loading behavior with if statements. For example, if isinstance(G, nn.DataParallel): torch.save(G.module.state_dict(), model_save_name) else: torch.save(G.state_dict(), model_save_…