I’m having an issue with properly deleting PyTorch objects from memory. With this Tensor:
test = torch.Tensor(1000,1000)
Then delete the object:
CUDA memory is not freed up.
Is there a clean way to delete a PyTorch object from CUDA memory?
It is because the cuda backend uses a caching allocator. This means that the memory is freed but not returned to the device.
if after running
del test you allocate more memory with
test2 = torch.Tensor(1000,1000), you will see that the memory usage will stay exactly the same: it did not re-allocated memory but re-used the one that had been freed when you ran
Ah Thanks a lot! It’s really helpful!
Which function do you use to monitor the gpu memory usage?
You can find them in the doc here.
I am using different models for inference. But my gpu space is only 10gb.
How do I unload a model from cuda and switch/load another model to cuda?
model.cpu() will move it back to the cpu.
Assuming that nothing else references the weights, they will be freed and returned to our allocator to be used for other Tensors.