nn.DataParallel with GANs

Hello all,

I have a question regarding the usage of DataParallel:

model = nn.DataParallel(model)

When working with GANs, I use the above line where model is replaced by the Generator and Critic respectively. However, except for the opportunity to use a higher batch size as there is more memory, there is no actual speedup for the training process.
In fact, it is slower then training the whole network on a single GPU.
Is this expected when working with generative adversarial networks or networks in general?
The GPUs that I am running on are the same, so I do not see the possibility of a bottleneck by a slower GPU.

Thanks in advance.

Depending on the workload you are dealing with, you might see the communication overhead using nn.DataParallel.
Could you post some information on the models and data (batch size) you are currently using?

The network architecture that I am using is this one: https://github.com/bluestyle97/STGAN-pytorch
It is a port of STGAN to PyTorch.
When training on one GPU, I use a batch size of 32. When training on 2 GPUs, I use a batch size of 64, as to still keep 32 samples on each GPU.
I know that the speedup will not be by a factor of 2, however, the network is slower by a factor of 4, compared to running it on a single GPU.