Is there a memory leak happening?

1

Greetings. I have a question about the GPU memory usage in the PyTorch.
When I ran the GAT model using pytorch and torch_geometric, I countered the cuda out-of-memory problem every time.
Although I use the large knowledge graph (105.43MB) and the 9 types of datasets for the training, validation, and test, I think it is weird.
Is it normal for memory usage to increase with each epoch?
Please check the memory usage in the screenshot.

After the first epoch
image

After the second epoch
image

After the third epoch
image

After the fifth epoch
image

If it is a memory leak, what part of my code should I look at?
Thank you for reading the question.

2

The memory usage goes down in the third epoch which does not point towards a memory leak.

1 Like
3

Thank you for your reply! I’ll try other approaches to reduce the memory usage.