Extremely high CPU cost after running multi-GPU program

1

Hi all,

Don’t know if you have encountered this issue. I installed the latest pytorch compiled from source. However, after I run a model loaded in multiple GPU, the CPU cost is extremely high, which takes almost 30 times more than usual, which limits my GPU server to run more training experiments.

However, such a phenomenon is not observed while I installed the binary pytorch version.

One more issue, I also noticed the pytorch compiled source also consumes more GPU memory. My model compilation contains a for loop, which doesn’t take much computational cost, but after I use this pytorch version, the gpu memory increases dramatically, and I have to set a very small batch size.

If anyone ever encountered this problem and is able to solve it. Pls let me know.

Thank you very much.

2

I fixed the GPU memory by installing the cudnn in anaconda. Compiling the pytorch from source doesn’t install cudnn either, which needs us to further install it by our own.

However, the high CPU performance issue still exists.