Does cuda cache memory for future usage?

Uncategorized
2

  1. The majority of the mentioned used memory comes from the creation of the CUDA context, which cannot be freed. You should also be able to see this memory usage by creating an empty CUDATensor.
    The caching allocator will reserve some memory which will be used later and you cannot disable it at the moment. Using the caching allocator will avoid synchronizing memory allocations during the model execution and should therefore speedup the execution. To check the cached memory, you could use print(torch.cuda.memory_reserved()). This post has a small example to show the difference between allocated and reserved memory.

  2. Yes, no_grad() will avoid storing intermediate gradients, which would be needed during the backward pass to calculate the gradients. The memory difference of the small tensors in your example might not change the used memory in nvidia-smi, so you could increase the size and use torch.cuda.memory_allocated to check the difference.