model.to(device) or nn.DataParallel(model) moves the model to the GPU but is there a way to check if the particular device is already in use by any other process.
The actual problem is when some else is training something on the server I don’t want my program to disturb their process and my program to fail so that I can allocate some other resource.
Unfortunately it is very hard to do a general check like this as it will depend a lot on how your manage your cluster with coworkers.
You can use nvidia-smi to check memory usage and/or process running on the different GPUs potentially.
You can check this small tool for example on how to do such thing and adapt it to your needs.
1 Like
nvidia-smi is more like a manual job.
waitGPU is very helpful will little changes it should solve my problem.
Thanks a lot.