The training hangs without printing any logs. Observations/configurations: 4 nodes. 4 GPU/node. distributed training with each process taking 1 GPU. pytorch version = 1.1; cuda version = 9.0; gpu driver version: 410.78 use the code base of facebook/maskrcnn-benchmark, but i thought it is just norm…

Distributed training hangs

gemfield (Gemfield) September 3, 2020, 1:04pm 3

Distributed data parallel freezes without error message

Env: Ubuntu 18.04 Pytorch 1.6.0 CUDA 10.1 Actually, I am using Docker image gemfield/pytorch:1.6.0-devel which stated in https://github.com/DeepVAC/deepvac (same with above env), and use PyTorch DDP (by use the class DeepvacDDP in https://github.com/DeepVAC/deepvac/blob/master/deepvac/syszux_deepvac.py) to train my model, which the code worked perfect yesterday. But today when I launch the train program again, the DDP is stucked in loss.backward()， with cpu 100% and GPU 100%。 There has no co…

show post in topic

Home
Categories
FAQ/Guidelines
Terms of Service
Privacy Policy

Powered by Discourse, best viewed with JavaScript enabled