Loss not decreasing using nn.DataParallel

1

Hi,
I am using nn.DataParallel on 4 GTX 1080 gpus, with

net = Net().cuda()
net = nn.DataParallel(net)
optimizer = torch.optim.Adam(net.parameters())
criterion = nn.CrossEntropyLoss().cuda()

for training

pred = net(data)
loss = criterion(pred,label)
optimizer = zero_grad()
loss.backward()
optimizer.step()

In this case, the training and validation loss stay around 0.4. While training on a single 1080ti gpu, both loss can reduce to 0.05.

Does anyone know what’s wrong? Any suggestion is appreciated.