Multiple gpu training problem

1

Recently I tried to train models in parallel using multiple GPUs (4 gpus). But the code always turns dead and the GPU situation is like this:

635E9264-74E8-4287-BE8F-3DEC05756A7B
635E9264-74E8-4287-BE8F-3DEC05756A7B800×336 453 KB

More specifically, when using only 2 gpus it works well. But when changing to more than 2 gpus, it doesn’t work at all…I’m not sure what’s going on and could someone give me some advices on that?

2

Solved, after updating the pytorch to the latest version.

3

I use pip to install the newest version but the problem still exists. It just turns dead and nothing happens. I have to kill the process myself.