DataParallel not effective


(Qiao Jin) #1

Hi all,
I have a quite large model and need to do data parallel among multiple GPUs.
I used:

model = nn.DataParallel(model)

And there are three visible GPUs. The GPU usage is:

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     12444      C   python                                     10741MiB |
|    1     12444      C   python                                      4663MiB |
|    2     12444      C   python                                      4633MiB |
+-----------------------------------------------------------------------------+

GPU1 and GPU2 are not fully utilized, but I cannot increase the batch size because then there will be memory error on GPU0.
Does anyone know how to solve this problem?