Cuda runtime error (60): peer mapping resources exhausted at

1

pytorch version:1.1.0
cuda version:10.0.130
GPU: TITAN V

I met a problem when I use 10 GPUs:

RuntimeError: cuda runtime error (60) : peer mapping resources exhausted at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/THC/THCGeneral.cpp:164

And when I use 8 or 4 GPUs，I didn’t get this error.

1 Like
2

I am having the same problem with 10 GPUs, #GPU <= 8 is file.
pytorch version:1.2.0
CUDA Version 10.1.105

1 Like