Something wrong when i tried to use two gpus in a node

I use the command line below to run the script(pytorch example of dali)

python -m torch.distributed.launch --nproc_per_node=1 -t

when the nproc_per_node=1,it work. But when nproc_per_node=2,there is a error.

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
=> creating model 'resnet50'
=> creating model 'resnet50'
Traceback (most recent call last):
  File "/home/zyy/anaconda3/envs/python36/lib/python3.6/", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/zyy/anaconda3/envs/python36/lib/python3.6/", line 85, in _run_code
    exec(code, run_globals)
  File "/home/zyy/anaconda3/envs/python36/lib/python3.6/site-packages/torch/distributed/", line 246, in <module>
  File "/home/zyy/anaconda3/envs/python36/lib/python3.6/site-packages/torch/distributed/", line 242, in main
subprocess.CalledProcessError: Command '['/home/zyy/anaconda3/envs/python36/bin/python', '-u', '', '--local_rank=1', '-t']' died with <Signals.SIGSEGV: 11>.

Hi @zyyupup,

Can you try this please:

python -m torch.distributed.launch --nproc_per_node=2 -a resnet50 --dali_cpu --fp16 --b 32 --static-loss-scale 128.0 --workers 4 --lr=0.4 ./ 2>&1

And give us full repro step and your environment (Anaconda, PyTorch, DALI versions)

Thank you for your reply. I tried the commamd and the same error accured again. My environment is:

  • Anaconda(python):3.6.8
  • Pytorch:1.2.0+cuda9.2
  • DALI:0.13.0
  • System:ubuntu16.04 with 2 gpus

I only changed the path of the dataset in the code and then run it.

DALI example is based on an older version of PyTorch APEX example - You can try to run it as well to check if this may be the DALI fault.