I was given access to a remote workstations where I can use a GPU to train my model. Of course all the resources are shared and the GPU memory is often partially used by other people processes.
Does getting a CUDA out of memory error means I just made someone else’s process crash?
It’s possible but not sure.
Basically whichever user asks for memory when it is full will see its process getting killed.
So it can happen to only one user, or all of them if they all happen to ask for memory at that time.
1 Like
Thanks for the answer! Anyway, that’s too bad… I don’t want to kill someone else’s model that may be training for a lot and I don’t want my model killed for the same reason! Have you ever worked in such scenario? Do you know of any tricks to prevent it?
I don’t think there is a good solution. Even if you would limit your code to e.g. 25% of the device memory, other processes might still want to use the whole GPU and thus creating an OOM error. 