Minibatch size and multiple GPUs

vision
1

Does the minibatch size divides into number of GPUs when using nn.dataparallel?

I had the minibatch size of 16 and when I use nn.dataparallel with 4 GPUs,
the actual initial input shape which goes through the model (printed it in the beginning of foward function) was like this
[4, channel, width, height]
[4, channel, width, height]
[4, channel, width, height]
[4, channel, width, height]

Is this because of the minibatch size is divided into each 4 GPUs and compute parallel?

2

Yes, nn.DataParallel splits the input batch in dim0 into n_gpu chunks and sends each one to the corresponding device.