Error with multiple GPUs: runtime error: nccl error 2: unhandled system error

1

When I run my code with multiple GPUs, it crashes occasionally with the following error:

File "main.py", line 132, in train
    model.train(train_loader, val_loader)
  File "/mnt/DATA/code/bitbucket/drn_seg/segment/seg_model.py", line 54, in train
    self.train_epoch(epoch, train_loader, val_loader)
  File "/mnt/DATA/code/bitbucket/drn_seg/segment/seg_model.py", line 93, in train_epoch
    output = self.model_seg(image)[0]
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 113, in forward
    replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 118, in replicate
    return replicate(module, device_ids)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/replicate.py", line 12, in replicate
    param_copies = Broadcast.apply(devices, *params)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/_functions.py", line 17, in forward
    outputs = comm.broadcast_coalesced(inputs, ctx.target_gpus)
  File "/usr/local/lib/python3.5/dist-packages/torch/cuda/comm.py", line 40, in broadcast_coalesced
    return torch._C._broadcast_coalesced(tensors, devices, buffer_size)
RuntimeError: NCCL Error 2: unhandled system error
2

Hi could you figure out the problem?

3

Here are some steps you can take to troubleshoot this error:

Check that all GPUs are properly connected and recognized by the system. Use the nvidia-smi command to verify that all GPUs are visible and have sufficient memory.

Verify that the network connection between the GPUs is working properly. If the GPUs are connected through a network, check that the network settings are configured correctly and that there are no connectivity issues.

Check that there are no resource allocation issues, such as insufficient memory or disk space. Make sure that the software is not trying to use more resources than are available.

Upgrade to the latest version of NCCL and ensure that it is properly installed and configured.

If the error persists, try reducing the number of GPUs used to see if the error disappears. This can help isolate the source of the problem.

Consult the documentation of the software that you are using and the NCCL library to look for known issues and solutions.

Finally, if none of the above steps resolves the issue, consider seeking help from the community or the developers of the software you are using.

Regards,
Rachel Gomez