# \[W CudaIPCTypes.cpp:22\] Producer process has been terminated before all shared CUDA tensors released. See Note \[Sharing CUDA tensors\] **URL:** \ **Category:** Uncategorized\ **Created:** [June 18, 2021, 1:28pm UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445 "2021-06-18T13:28:49Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![B.W\_Zhang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/b.w_zhang/32/29590_2.png) [@B.W\_Zhang](https://discuss.pytorch.org/u/B.W_Zhang)\ **Post date:** [June 18, 2021, 1:28pm UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/1 "2021-06-18T13:28:49Z")
I got this warning at the end of each epoch when using multiple GPUs: `[W CudaIPCTypes.cpp:22] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]` But it doesn’t seem to affect the training since the result is as good as it is. But I would still like to know what the probable cause is and how to solve it 😀 I simply use: `model = torch.nn.DataParallel(model)` to enable multi-GPUs. Besides, I also added `torch.multiprocessing.set_start_method('spawn', force=True)` in my code. Don’t know whether it has any effect on this. Thank you for the answer in advance. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [June 19, 2021, 2:07am UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/2 "2021-06-19T02:07:33Z")
Are you seeing the same message, if you remove the multiprocessing code (I assume this is triggering it)? ---
**Author:** ![B.W\_Zhang](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/b.w_zhang/32/29590_2.png) [@B.W\_Zhang](https://discuss.pytorch.org/u/B.W_Zhang)\ **Post date:** [June 23, 2021, 9:44am UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/3 "2021-06-23T09:44:41Z")
Sorry for my late response and thank you for the reply! I pinned the trigger of this warning. It happens if I set num\_workers \> 0 in torch.utils.data.DataLoader, instead of being caused by nn.DataParallel(). Any idea how to fix it? Is it probably due to that I set `torch.multiprocessing.set_start_method('spawn', force=True)`? ---
**Author:** ![Ilya\_Kotlov](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ilya_kotlov/32/35840_2.png) [@Ilya\_Kotlov](https://discuss.pytorch.org/u/Ilya_Kotlov)\ **Post date:** [July 15, 2021, 11:03pm UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/4 "2021-07-15T23:03:19Z")
I have issued the same problem when setting num of workers to more then 0. @ptrblck, any solution ? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [July 16, 2021, 5:48am UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/5 "2021-07-16T05:48:56Z")
Unfortunately, I don’t have suggestions, as I don’t fully understand the use case of using `torch.multiprocessing` as well as multiple workers (which itself will use multiple processes), so could you explain the use case a bit more? ---
**Author:** ![visionscaper](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/visionscaper/32/44736_2.png) [@visionscaper](https://discuss.pytorch.org/u/visionscaper)\ **Post date:** [December 8, 2021, 10:46pm UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/6 "2021-12-08T22:46:48Z")
Hi @ptrblck, I came across the same issue and was intrigued by your remark: > [@ptrblck](#): > > I don’t fully understand the use case of using `torch.multiprocessing` as well as multiple workers For one, isn’t it good to be able to process/load the data using a separate process, apart from the training process(es), in order to not tax them with extra processing load? Further, although it’s not my use case, loading the data might involve some heavy preprocessing, warranting even more than one data loader workers per training process; e.g. in the case in of some vision applications. Maybe you have a completely different idea about this and give us your insight in to this topic. Thanks in advance. ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [December 9, 2021, 4:33am UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/7 "2021-12-09T04:33:12Z")
I totally agree with you and also think that multiple processes by themselves are useful, should be used for the data loading and process, and would yield a speedup in the overall training pipeline. This can be easily achieved with the `num_workers` argument of the `DataLoader`. My comment might not have been clear enough, but I was wondering about the use case to use the `multiprocessing` package manually in the training script as well as multiple workers (which will again spawn multiple processes, so you would end up with a “nested” multiprocessing workload), as I would imagine that using the simpler approach of setting `num_workers>=1` should already do the job. ---
**Author:** ![visionscaper](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/visionscaper/32/44736_2.png) [@visionscaper](https://discuss.pytorch.org/u/visionscaper)\ **Post date:** [December 9, 2021, 11:55am UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/8 "2021-12-09T11:55:55Z")
Ah, ok, sorry, I didn’t notice the part about using the `multiprocessing` package separately. I think this is because I had the same issue just using DDP and Dataloaders with num\_workers \> 0. I get these errors just before my script exits; I already am a good citizen by using `dist.barrier()` to wait for all training processes to complete before exiting. When I set the `num_workers` to zero the script exits smoothly without any errors. In the past I have often used multiple DataLoader workers in a DDP context and never had this issue. I’m wondering if this is something that sneaked in to a recent Pytorch release or is related to recent a CUDA version? My setup: ```auto | NVIDIA-SMI 470.86 Driver Version: 470.86 CUDA Version: 11.4 | ``` ```auto $ pip show torch Name: torch Version: 1.10.0 ``` This is on a 4x V100 machine with Ubuntu. It might also be in my script of course. ---
**Author:** ![visionscaper](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/visionscaper/32/44736_2.png) [@visionscaper](https://discuss.pytorch.org/u/visionscaper)\ **Post date:** [December 9, 2021, 12:03pm UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/9 "2021-12-09T12:03:29Z")
Ok, I dove a bit more in to this, and found the issue. In this particular instance I am using a model library where the author unfortunately had added the batch collate function as a method to the model class. Because of this the model `self` is also transported to the DataLoader workers. When the training scripts exits the DataLoader worker processes are killed too, without properly dealing with the Tensors that are copied to the worker. I refactored all the batch collating code out of the model class and thus made it independent of the `self` of the model instance. This resolved the issue, exiting the script is very smooth now! So, if anyone gets this error, it could be because your are copying the model over to the DataLoader workers without knowing it. ---
**Author:** ![sjtuhl](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sjtuhl/32/47143_2.png) [@sjtuhl](https://discuss.pytorch.org/u/sjtuhl)\ **Post date:** [March 1, 2022, 8:54pm UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/10 "2022-03-01T20:54:58Z")
Hi @visionscaper, I’m encountering the same issue. I’m currently using pytorch lightning to write my model. When I set the num\_worker \> 0 in the dataloader, the warning will occur. I’m a beginner in pytorch. Could you give me some example that how did you refactor your batch collating code out of model class? I don’t know how to do it. ---
**Author:** ![visionscaper](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/visionscaper/32/44736_2.png) [@visionscaper](https://discuss.pytorch.org/u/visionscaper)\ **Post date:** [March 2, 2022, 7:00pm UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/11 "2022-03-02T19:00:39Z")
Hi @sjtuhl, if you have a similar issue to mine, somehow your model is linked to your `dataset`, `collate_fn` or other object given to a `DataLoader`. Because of this, when you set `num_worker > 0`, the model is copied to the `DataLoader` workers. For instance, if your `dataset` is a property of your model class instance (which is just bad design), not only your dataset is copied to the workers, but everything it is “attached” to as well. I hope this makes sense to you. I can’t really help you without inspecting your code. ---
**Author:** ![Eveefun](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eveefun/32/64126_2.png) [@Eveefun](https://discuss.pytorch.org/u/Eveefun)\ **Post date:** [September 22, 2023, 4:20am UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/13 "2023-09-22T04:20:20Z")
Hi @visionscaper, I have the same issue as you. My data collate fn is written inside the dataset model. There are some tensors my collate\_fn need, stored in the dataset model. I wonder if there is a way to avoid moving the collate\_fn out and escape the error? 🤗 ---
**Author:** ![Garret\_Kaye](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/garret_kaye/32/72135_2.png) [@Garret\_Kaye](https://discuss.pytorch.org/u/Garret_Kaye)\ **Post date:** [August 15, 2024, 6:02am UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/14 "2024-08-15T06:02:40Z")
This warning is caused by tensors (or other objects) that are not being properly cleaned up (released) on the CUDA device before the process they belonged to was terminated. As it was for me in my case, and seems to be the case for mostly everyone else, the culprit behind this warning has to do with moving objects to, or owning objects on, the CUDA device from inside of one of the `DataLoader` worker processes, and the worker process being terminated before the objects are released from CUDA. Moving tensors to CUDA inside a `collate_fn` that is passed to a `DataLoader` where `num_workers > 0` - which just means that a child process will be created for each worker - is an example of this, and is exactly what I was doing. In my training loop then, after all the training samples were iterated through, my `DataLoader` worker processes would be terminated and I would get this warning. As tempting as it is to move batches to CUDA inside the `collate_fn`, and though it may even provide some performance gains, the potential for a memory leak or some other resource management issue along with the annoying warning made me rethink my strategy. I ended up just relocating this logic to the first line in my training loop (on CPU - main process) where I move each batch to CUDA before my forward pass which I think is generally recommended anyway - never looked back since. Alternatively, if you really wanted to, you could keep whatever objects you want on the CUDA device inside your `DataLoader` worker processes if you set `persistent_workers=True` in your `DataLoader` constructor and not encounter this warning until your `DataLoader` is entirely deallocated. You may encounter other issues though like running out of memory on the GPU, but I think for some use cases, maybe doing this could be beneficial. Hope this helps. ---
**Author:** ![zsxlkn](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zsxlkn/32/75919_2.png) [@zsxlkn](https://discuss.pytorch.org/u/zsxlkn)\ **Post date:** [February 28, 2025, 8:54am UTC](https://discuss.pytorch.org/t/w-cudaipctypes-cpp-22-producer-process-has-been-terminated-before-all-shared-cuda-tensors-released-see-note-sharing-cuda-tensors/124445/16 "2025-02-28T08:54:17Z")
Very helpful,Thank you.The issue has been solved.