Memory leak in Pytorch 1.10

vision
1

I run the following code Ind I have a memory leak

from time import sleep
import cv2
from torch import from_numpy, load as torch_load, cuda

def predict(image):  # numpy image
        print(f'memory before: {cuda.memory_stats()}')
        x_tensor = from_numpy(image).to(DEVICE).unsqueeze(0)
        del x_tensor
        sleep(20)
        cuda.empty_cache()
        sleep(20)
        cuda.synchronize()
        sleep(20)
        cuda.empty_cache()
        sleep(20)
        cuda.reset_max_memory_allocated()
        sleep(20)
        print(f'memory after: {cuda.memory_stats()}')


image = cv2.imread('my_image.png')
for i in range(10):
    predict(my_image)

The output is like for ‘active.all.allocated’ key

memory before: 0
memory after: 1
memory before: 1
memory after: 2
…
memory before: 9
memory after: 10

Is thre a workaround for the given version? It would be time consuming to retrain all the models and the models are incompatible with a new smp/PyTorch.
What can cause this memory leak? Is it an old bug that is fixed in more recent versions of PyTorch?

2

Using print(f'summary: {cuda.memory_summary()}') shows the expected results of 0B in the current usage and an increased total allocation count.

3

Thank you for your response.
So, how do I check for a memory leak? Situation. When I analyze some 10 photos one by one it works fine. Then the application fails with memory allocation error. I do shrink all photos to 10 megapixel size and I do know that my GPU can hold and process 10- Mpixel photos.
But there is a clear memory leak buildup. Possibly some memory fragmentation issues.
I do

del tensor
cuda.empty_cache()

after each analysis.
What can cause the problem and how to fix it?
PS We are working on the analysis of skin health if it helps.

PPS. In the actual application I load my models (takes some 2 Gb out of 8 Gbs). Then I receive and analize photos. Each photo is shrunk to 10- Mpixels prior to analysis. Works fine on the first ten photos, then fails with

RuntimeError: CUDA out of memory. 
Tried to allocate 1.17 GiB (GPU 0; 7.93 GiB total capacity;
3.47 GiB already allocated; 
1.03 GiB free; 6.14 GiB reserved in total by PyTorch) 
If reserved memory is >> allocated memory try setting 
max_split_size_mb to avoid fragmentation.  
See documentation for Memory Management and 
PYTORCH_CUDA_ALLOC_CONF
4

Can you please expand your explaination?
Do I have a memory leak or not?

When I run my app and analize the same photo hundred times it crashes with “out of memory. Tried to allocate 1.2 Gb of memory, only have 1.1 free memory”.