How to swap data between cpu and gpu

1

hi,
How to swap part of a model data including the parameters tensor and intermediates results tensor to host memory(cpu) while free the GPU memory they occupied in pytorch?

2

Why is .cpu() method not sufficient?

3

From the docs, .cpu() returns a copy of gpu data, does it free the occupied GPU memory?

4

If you want to release the GPU memory, use torch.cuda.empty_cache().
I found a topic about how to release a tensor here, but I’m not sure it will work…

5

thanks, seems to be a solution.

6

Note that this approach will free the GPU memory indeed, so that other applications can use it.
However, if you want to use the memory again in PyTorch, it would have to be reallocated, which might slow down your code. To avoid it, PyTorch uses a custom caching mechanism, so that the cached GPU memory can easily be reused without any allocations.

1 Like