CUDA device error: CUDA-capable device is/are busy

Has anyone met this problem before? I ordered 2 GPU on one node

What does “I ordered 2 GPUs” mean? Are you using a shared node? If so, is another user blocking the devices? Are you able to use both devices standalone?

Hi, they are on a shared node.

In that case the error is expected since all your GPUs are in Exclusive Process compute mode.

Sorry for the late reply and thank you so much for your suggestion!