I’m unsure why you want to use a single device for the validation step only, but I would not move the DDP model to this device. Instead you could try to access the internal .module attribute in the process running on GPU3 and perform the validation step. To do so, check which rank you are currently on and don’t move the module to GPU3 from other ranks.