Why i use multi_gpu to train my model the generated image became deteriorated

I use 2 gpu to train my model, and i watch the gpu state, there were 2 gpu running, but the generate image became deteriorated, anyone can help me,thank you!

It would help if you’d provide more information in your post. Like what model are you training and for what purpose. Does it work with one gpu?

this is the way i use the dataparallel function,

,i am sure there were 2 gpus working when i run the code, and it works with one gpu, when i use large image ,one gpu is out of memory ,so i want to use 2 gpu,but the generated image became deteriorated ,i don not know why?

Ok so it’s a cycleGAN and you want to do image-to-image translation. Does 2 gpus really help you feed larger images into the network or simply allows you to increase the batch size? What image sizes are you working with?

it is the another form of cyclegan,named augment cyclegan, the size of image in this code is 6464 and transform it to numpy data, where i want to change the size to 128128, one gpu is not work,so i use 2 gpus,i show the GPU state in wanch -n 1 nvidia-smi, and there were 2 gpus working, but the generated image is much deteriorated than use one gpu(the image size is 64*64 ) ,here are the generated image,one is use one gpu,the other is use 2 gpus.12 11

Yes that is a large difference. What batch size are you using? When you are using 2 GPUs the batch is split onto the different GPUs which can cause a problem if the network contains batch norm layers.
If you lower the batch size you can try with one GPU to see if that works with the 128x128 image before moving on to two GPUs.

Other than that, GANs have had problems with high resolution images before. Perhaps you can train the network on 64x64, save the weights and then retrain those weights on 128x128?

I’d also recommend some sort of system to check the loss + gradients to see that the network doesn’t suffer from gradient explosion. That can cause some very bad images. Have a look at tensorboard(x) and visdom.

Goood luck :smiley: