Using torch.nn.DataParallel with single GPU, everything works fine.
os.environ["CUDA_VISIBLE_DEVICES"] = '0'
model = resnet34()
model = Tripletnet(model)
model = torch.nn.DataParallel(model).cuda()
But when I switch to DistributedDataParallel with multi-gpu, I get an error
model = resnet34()
model = Tripletnet(model)
os.environ["CUDA_VISIBLE_DEVICES"] = '0,1,2,3'
model = torch.nn.parallel.DistributedDataParallel(model.cuda(), device_ids=[0,1,2,3])
File “/home/slrum/code03/main02.py”, line 288, in train
loss.backward()
File “/opt/anaconda3/lib/python3.6/site-packages/torch/tensor.py”, line 102, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File “/opt/anaconda3/lib/python3.6/site-packages/torch/autograd/init.py”, line 90, in backward
allow_unreachable=True) # allow_unreachable flag
File “/opt/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/distributed.py”, line 445, in distributed_data_parallel_hook
self._queue_reduction(bucket_idx)
File “/opt/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/distributed.py”, line 475, in _queue_reduction
self.device_ids)
TypeError: _queue_reduction(): incompatible function arguments. The following argument types are supported:
1. (process_group: torch.distributed.ProcessGroup, grads_batch: List[List[at::Tensor]], devices: List[int]) → Tuple[torch.dist
ributed.Work, at::Tensor]Invoked with: <torch.distributed.ProcessGroupNCCL object at 0x2b9afa20d3b0>, [[None, tensor([[[[-1.2661e-04, 2.7497e-04, 1.3505e-
03, …, 4.6058e-03, 3.8858e-03, 4.0022e-03],
[ 1.4730e-03, 2.2759e-03, 5.1621e-03, …, 4.3713e-03,
8.9991e-04, 2.5457e-03],
What does this error mean? How could I fix it?