CUDA out of memory How to fix?

1

How to clear GPU cache
torch.cuda.empty_cache() doesn’t work

2

The first question I would ask is the number of GPU cores I have.
For instance if I have 8 cores, then torch.cuda.empty_cache() should be rephrased as something like this:

how_many_gpus = torch.cuda.device_count()
    for _ in range(how_many_gpus):
        torch.cuda.set_device(_)
        torch.cuda.empty_cache()

What have you tried so far?

3

I have NVIDIA GeForce GTX 1060 6GB

4

.empty_cache will only clear the cache, if no references are stored anymore to any of the data. If you don’t see any memory release after the call, you would have to delete some tensors before.

5

How can I clear cache? I am using this https://github.com/avinashpaliwal/Super-SloMo

6

I tried this and
TypeError: device_count() takes 0 positional arguments but 1 was given

7

What is wrong with this

import torch
how_many_gpus = torch.cuda.device_count()
for _ in range(how_many_gpus):
        torch.cuda.set_device("cuda0")
        torch.cuda.empty_cache()
8

Have you meant to say you don’t have multiple GPUs? Can you confirm?

9

Please check out the CUDA semantics document.

Instead, torch.cuda.set_device("cuda0") I would use torch.cuda.set_device("cuda:0"), but in general the code you provided in your last update @Mr_Tajniak would not work for the case of multiple GPUs.

In case you have a single GPU (the case I would assume) based on your hardware, what @ptrblck said:

.empty_cache will only clear the cache, if no references are stored anymore to any of the data. If you don’t see any memory release after the call, you would have to delete some tensors before.

This basically means PyTorch torch.cuda.empty_cache() would clear the PyTorch cache area inside the GPU.

You can check out the size of this area with this code:

import torch

import gc
def p():
    c = torch.cuda.memory_cached()
    print(f'cached   :{c}')
    a = torch.cuda.memory_allocated()
    print(f'allocated:{a}')
    f = torch.cuda.memory_cached()-torch.cuda.memory_allocated()
    print(f'free     :{f}')
    
torch.cuda.empty_cache()
p()
r = torch.randn(1, 128).cuda()
p()

Out:

cached   :0
allocated:0
free     :0
cached   :2097152
allocated:512
free     :2096640

See how PyTorch allocated 2Mb of cache just for storing this 128 floats. If you would del r followed by p() the GPU memory will be free again.

If you would have some objects you haven’t deleted make sure you delete them if they are not needed.

Why did PyTorch cached the memory in advance?
To reuse it later. This is the idea of the cache. We assume this precious resource will be used later.

10

I have single GPU
GeForce GTX 1060 6GB

11

to free the GPU cache I have to enter

import torch
p()
del r

Yes?

12

@Mr_Tajniak, all I wanted to say you would need to deal with the variables that consume GPU memory smart.

I haven’t tested project you mentioned, but I just saw you created the issue.

You may expect the feedback in there. Check if OOM is there if you use a smaller video format. Check what are the GPU memory requirements based on the video format you use (size of the frame).

13

ok thanks for help. In case of problems I will contact you, ok?

14

In case of problems I will contact you, ok?

Sure. Why not.