How totally remove data and free gpu

I have a agile algorithm that use a changing amount of GPUs, increasing and decreasing the number of GPUs that are needs.It constantly checked for free GPUs, but it seems that I can not really free a GPU after I already sent an object to the GPU.
For example:

import torch
tm = torch.Tensor([1,2]).to("cuda")

|   0  GeForce RTX 208...  On   | 00000000:3D:00.0 Off |                  N/A |
|  0%   37C    P2    52W / 250W |    730MiB / 10989MiB |      0%      Default

So I use 730MiB…
Now no matter what I try I can not make the 730MiB go to zero:

del tm                                                                                                                                                                                                 
import sys;sys.modules[__name__].__dict__.clear()                                                                                                                                                      
Once deleted, variables cannot be recovered. Proceed (y/[n])? y
|   0  GeForce RTX 208...  On   | 00000000:3D:00.0 Off |                  N/A |
|  0%   35C    P8     1W / 250W |    728MiB / 10989MiB |      0%      Default |

Any help?


I’m afraid this is the cuda driver memory use.
As you can see in your first example: putting (almost) nothing on the GPU does the same.
There is no way to clear that driver’s memory I’m afraid unless you stop your process :confused:

Thanks @albanD I was sure it is possible. Damm… It completely change the way the algorithm works… Thanks.
I’ve opened a ticket -

We’ll see what the cuda experts say. But this is most likely almost impossible to do :confused:

I’ve checked with TensorFlow and there is a workaround -

But it is an annoying one… not sure how it would work with pytorch

This will work on pytorch as well: run your code in a different process. When this process finishes, it will free up all the GPU memory.