NCCL backend hangs when training over infiniband

distributed
#1

I train my model across several machines. I have two machines which have gpus and infiniband cards. The networks is 1Gbit, Infiniband is 2x40Gbit. When I remove cards, and start training everything works, though slower than on one machine. When I run with infiniband setup, the system just hangs. There’s 100% GPU utilisation, wattage is 1/2 of maximum, and there’s very little network activity.

Do you have any hints on how to proceed with finding out what’s wrong with training?

#2

It might be a bios issue. I have a similar issue with 4Xv100 GPU that runs too slow. It turns out to be a bios setup problem. There is a bios adjustments which needs to be arranged as performance mode.
[V100 is too slow for training](http://v100 topic)

#3

@otutay In your case it’s just too slow. In my case it plainly hangs.