Dynamically Changing Pluggable CUDA allocator

1

Hi, I came across a comment [1] suggesting that it’s possible to change CUDA memory allocators dynamically in different parts of an application. However, the same thread warns that it could be unsafe to do so once an allocator has been utilized. I find these messages a bit contradictory.

I have a specific use case that involves using PyTorch’s default CUDA memory allocator for model parameters and activations, and an external CUDA memory allocator (RMM) for the optimizer state. I am wondering if this is both possible and safe to do?

Thanks in advance for your time!

[1] Add Pluggable CUDA allocator backend by emcastillo · Pull Request #86786 · pytorch/pytorch · GitHub

2

What’s the reason for mixing both allocators? Do you see a disadvantage in using only one of them?

3

My motivation is to use external memory allocators to allocate managed memory (cudaMallocManaged), so they are pageable when oversubscribed.

Is it possible to designate certain parameters as pageable and others as non-pageable without combining allocators?