# PyTorch causes zombie processes on multi-GPU system

**URL:** <https://discuss.pytorch.org/t/pytorch-causes-zombie-processes-on-multi-gpu-system/140097>\
**Category:** Uncategorized\
**Created:** [December 23, 2021, 12:45pm UTC](https://discuss.pytorch.org/t/pytorch-causes-zombie-processes-on-multi-gpu-system/140097 "2021-12-23T12:45:41Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ATAboukhadra](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ataboukhadra/32/50067_2.png) [@ATAboukhadra](https://discuss.pytorch.org/u/ATAboukhadra)\
**Post date:** [December 23, 2021, 12:45pm UTC](https://discuss.pytorch.org/t/pytorch-causes-zombie-processes-on-multi-gpu-system/140097/1 "2021-12-23T12:45:41Z")

</div>

Hi,

I’m running my PyTorch code on a GPU on a remote server. The server has 8 GPUs and they are shared among multiple users. So I run my code on 1 GPU out of them using DataParallel class. Whenever I try to terminate the process it turns into a zombie process as you can see in the nvidia-smi output here:

 ![Screen Shot 2021-12-23 at 1.38.18 PM](https://discuss.pytorch.org/uploads/default/original/3X/4/1/41efdf0b4d6f7acab7ffd509c649b7b84214a91f.png)

All processes with a dash as the process name are killed processes that are still occupying the GPU and can’t be removed unless the system is rebooted which affects all users of the system.

I can’t produce this problem on my machine that has only 1 GPU and the code runs normally with no problems and terminates properly. This problem occurred on the multi-GPU system just last week and I’m not sure whether this was caused by a change in the PyTorch library or is a problem from my side.

If you can give me a fix or check what the problem might be it would be great.

Kind regards,  
Ahmed

---

<div class="post-metadata">

**Author:** ![VFAndy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vfandy/32/44730_2.png) [@VFAndy](https://discuss.pytorch.org/u/VFAndy)\
**Post date:** [January 5, 2022, 12:05pm UTC](https://discuss.pytorch.org/t/pytorch-causes-zombie-processes-on-multi-gpu-system/140097/2 "2022-01-05T12:05:06Z")

</div>

Running into the same issue here, Pytorch creates Deadlocking processes (I debugged into it, let me know if you want details where the dealock happens – seems to be a threading issues imo) which become zombies and use up a lot of GPU RAM while at the same time blocking any further or current GPU jobs. I will try adding the details (Pytorch version, cuda version etc.) later.