torch.cuda.set_per_process_memory_fraction() doesn’t see shared memory. I have a GPU with 8GB of video memory. I’m running a model that consumes my 8GB and overflows another 8GB into Windows’ shared video memory. When I set torch.cuda.set_per_process_memory_fraction(fraction=1.0, device=0), the process limits the total to just the GPU’s 8GB of video memory. I wanted to simulate, for example, telling PyTorch that I have, for example, 13GB available, also considering shared memory. I want to set a limit to know how far my model can load. For example, sometimes the model shows that it consumed 16.9GB considering shared memory. But I wanted to set it to 16GB to see if it would run because I have the impression that it doesn’t actually need the 16.9GB, but I can’t run this simulation.
PyTorch does not enable or use Windows’ “shared video memory” feature as it’s a driver feature. It thus also does not have any control when and if it will be used.
So you’re telling me that when PyTorch applies set_per_process_memory_fraction it’s only seeing the 8GB of my GPU. I tried using max_split_size_mb to set a limit, but that doesn’t stop the driver from allocating shared memory, even though I set it to something. It seems to only be related to the cache. So, there’s no way to tell PyTorch that you can use 8GB of the GPU and only 4GB of shared memory, right?
Yes, this option limits the allowed allocated memory by checking the total memory via cudaMemGetInfo as seen here. PyTorch does not have any knowledge about Windows driver features which offload GPU memory to the host behind your back without opting-in.