How to manually delete/free a tensor in ATen?

C++
1

I’m trying to build a custom C++ extension using the aten api.
Does aten library provide a simple way to manually delete/free a tensor? Please take a look at the example below:

// create a tensor
torch::Tensor tensor = torch::randn({3,4,5});
// manually delete this tensor
delete tensor;    // something like this

The target is to free the memory of some large tensors in a function before the function ends, in order to save the total memory usage and avoid ‘CUDA out of memory’.

I understand there might be some tricks like:

// option 1
torch::Tensor * tensor = torch::Tensor new(torch::randn({3,4,5}));
delete tensor;

and

// option 2
torch::Tensor tensor = torch::randn({3,4,5});
// free memory usage by setting it smaller
tensor.resize_(at::IntArrayRef{0});

But is there a simpler and more straightforward way to achieve this?

Note that I’ve already searched the forums and read this related question How to free GPU memory of at::Tensor in ATen, C++? but still got no answer.

Another question:
for an operation like

torch::Tensor a = a.clone();

will tensor a be automatically freed? I’m not sure whether this will cause memory leak.

1 Like
2

So I also added it to the other thread:

When a Tensor (or all Tensors referring to a memory block (a Storage)) goes out of scope, the memory goes back to the cache PyTorch keeps. You can free the memory from the cache using

#include <c10/cuda/CUDACachingAllocator.h>

and then calling

c10::cuda::CUDACachingAllocator::emptyCache();

(of course, you could try using torch:: instead of c10:: and see if it is automatically imported somewhere).

Best regards

Thomas

3

Hi Thomas,
Thanks a lot for you reply. I guess I might have not stated the problem clearly. My target is to free the memory of some large tensors manually in a function before these tensors are out of the function’s scope, in order to save the total memory usage and avoid “CUDA out of memory” error.
I have edited the problem description. Can you take a look again? Thanks!

4

Yeah well, I usually just introduce a local scope for that.

void myfunc() {
  some code here
  {
   torch::Tensor a = ... 
  }
  here a is gone
}

But if you must: Assigning the default-constructed undefined Tensor would likely work, too.

Best regards

Thomas

3 Likes
5

That helps! Thanks a lot.