Cuda out of memory & increasing memory usage

[RangoHU]

Hello,

When I ran my model on multiple GUPs, and ran into out of memory error. So I half the batch_size, and again ran into the same error (after a longer while).

So I checked the GPU memory usage with nivida-smi, and have two questions:
Here is the output of nivida-smi:

| 0 33446 C python 9446MiB |
| 1 33446 C python 5973MiB |
| 2 33446 C python 5973MiB |
| 3 33446 C python 5945MiB |
±----------------------------------------------------------------------------+

1. The memory on GPU_0 is roughly twice used than the other three. Is it nature that the first GPU is more used?

2. The memory usage increase slowly as the program goes, which eventually cause the out-memory-error. How can I solve the problem?

BTW, I installed pytorch using the latest Dockfile.

Thanks!

[Yifan]

Hi Dear RangoHU,

I face the same problem, do you have any idea why the memory increasing during training?

Thanks!

[Yifan]

My out of memory problem has been solved. Please check

CUDA memory continuously increases when net(images) called in every iteration

Hi, I have a very strange error, whereby, when I get by outputs = net(images) within every iteration in a for loop, the CUDA memory usage keeps on increasing, until the GPU runs out of memory. The weird situation is that if I have this loop inside a function, it causes this issue. If I just have the contents of the function sitting in my normal script, it works just fine. What may be the cause of this?? Thanks

[RangoHU]

Hi thanks, I checked this thread, but didn’t help in my case.