How totally remove data and free gpu

1

Hello,
I have a agile algorithm that use a changing amount of GPUs, increasing and decreasing the number of GPUs that are needs.It constantly checked for free GPUs, but it seems that I can not really free a GPU after I already sent an object to the GPU.
For example:

import torch
tm = torch.Tensor([1,2]).to("cuda")
!nvidia-smi

|===============================+======================+======================|
|   0  GeForce RTX 208...  On   | 00000000:3D:00.0 Off |                  N/A |
|  0%   37C    P2    52W / 250W |    730MiB / 10989MiB |      0%      Default

So I use 730MiB…
Now no matter what I try I can not make the 730MiB go to zero:

del tm                                                                                                                                                                                                 
torch.cuda.empty_cache()                                                                                                                                                                               
import sys;sys.modules[__name__].__dict__.clear()                                                                                                                                                      
%reset                                                                                                                                                                                                 
Once deleted, variables cannot be recovered. Proceed (y/[n])? y
!nvidia-smi
|   0  GeForce RTX 208...  On   | 00000000:3D:00.0 Off |                  N/A |
|  0%   35C    P8     1W / 250W |    728MiB / 10989MiB |      0%      Default |

Any help?
Thanks

2

Hi,

I’m afraid this is the cuda driver memory use.
As you can see in your first example: putting (almost) nothing on the GPU does the same.
There is no way to clear that driver’s memory I’m afraid unless you stop your process :confused:

3

Thanks @albanD I was sure it is possible. Damm… It completely change the way the algorithm works… Thanks.
I’ve opened a ticket - https://github.com/pytorch/pytorch/issues/37664

4

We’ll see what the cuda experts say. But this is most likely almost impossible to do :confused:

5

I’ve checked with TensorFlow and there is a workaround -


But it is an annoying one… not sure how it would work with pytorch

6

This will work on pytorch as well: run your code in a different process. When this process finishes, it will free up all the GPU memory.