# Errors when using num\_workers\>0 in DataLoader

**URL:** <https://discuss.pytorch.org/t/errors-when-using-num-workers-0-in-dataloader/97564>\
**Category:** Uncategorized\
**Created:** [September 26, 2020, 11:05am UTC](https://discuss.pytorch.org/t/errors-when-using-num-workers-0-in-dataloader/97564 "2020-09-26T11:05:44Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![simoloca](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@simoloca](https://discuss.pytorch.org/u/simoloca)\
**Post date:** [September 26, 2020, 11:05am UTC](https://discuss.pytorch.org/t/errors-when-using-num-workers-0-in-dataloader/97564/1 "2020-09-26T11:05:44Z")

</div>

Hi all, I’m facing a problem when setting the num\_workers value in the DataLoader bigger than 0.  
In particular I’m trying to train a custom model on a custom dataset.  
Thus, if I hold the num\_workers=0, everything it’s fine and the whole process is successful.  
But for every other configuration on num\_workers, the problem persist for every setting I try with the batch\_size used, number of epochs for traning, ecc.  
And it seems that when num\_workers is greater than 0, x for istance, the script try to run x times and then the error came out.

The strange issue comes when I try to run the script with the bottleneck utility with the num\_workers settings bigger then 0, and in this way it works correctly.  
So my question is if the bottleneck utility applies some type of optimization that I don’t do, or when we want to set the num\_workers bigger than 0 we need to so something in particular.

P.S.  
I want to use the num\_worker\>0 in order to push my GPU to the max usage.

Thanks.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bramvanroy/32/28628_2.png) [@BramVanroy](https://discuss.pytorch.org/u/BramVanroy)\
**Post date:** [September 26, 2020, 2:44pm UTC](https://discuss.pytorch.org/t/errors-when-using-num-workers-0-in-dataloader/97564/2 "2020-09-26T14:44:55Z")

</div>

Num\_workers sets the number of CPU workers in the data loader only. This has nothing to do with GPU utilization - although faster batch preprocessing will lead to batches being loaded faster and thus more streamlined GPU usage. On Windows, due to multiprocessing restrictions, setting num\_workers to \> 0. This is expected, so don’t worry about it. You can set it to 0.

---

<div class="post-metadata">

**Author:** ![simoloca](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@simoloca](https://discuss.pytorch.org/u/simoloca)\
**Post date:** [September 26, 2020, 3:23pm UTC](https://discuss.pytorch.org/t/errors-when-using-num-workers-0-in-dataloader/97564/3 "2020-09-26T15:23:52Z")

</div>

Thanks for the reply!  
But I saw that when I run the program with the bottleneck tool, where num\_workers is \> 0, the GPU “cuda usage” is much more efficient and stable, in the sense that it is always at about 95-100%.  
Instead, when I had to run the script normally without the bottleneck tool and num\_worker is 0, the GPU cuda usage is more discontinuous (it goes to 50%, to 90% and it turns to 25 and so on), unstable.  
So, this behaviour make me think that it has something to do with the gpu usage.

---

<div class="post-metadata">

**Author:** ![simoloca](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@simoloca](https://discuss.pytorch.org/u/simoloca)\
**Post date:** [September 26, 2020, 4:08pm UTC](https://discuss.pytorch.org/t/errors-when-using-num-workers-0-in-dataloader/97564/4 "2020-09-26T16:08:52Z")

</div>

Ok, it seems that I have solved the problem, and now I can set the num\_workers\>0.  
What I did was just to insert all the code that train the network in a function: train(), summaring:

```auto
def train():
    # Here was inserted the whole code that train the network ...
if __name__ == ' __main__':
    train()

```

Could anyone explain how and why this works?

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bramvanroy/32/28628_2.png) [@BramVanroy](https://discuss.pytorch.org/u/BramVanroy)\
**Post date:** [September 26, 2020, 4:36pm UTC](https://discuss.pytorch.org/t/errors-when-using-num-workers-0-in-dataloader/97564/5 "2020-09-26T16:36:09Z")

</div>

You can have a look here: [https://docs.python.org/3.7/library/multiprocessing.html#multiprocessing-programming](https://docs.python.org/3.7/library/multiprocessing.html#multiprocessing-programming)

Note that Windows can still cause issues here, be it pickling errors or slow performance. See [https://github.com/pytorch/pytorch/issues/12831](https://github.com/pytorch/pytorch/issues/12831). The safest approach is using 0 on Windows.

---

<div class="post-metadata">

**Author:** ![simoloca](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/ed655f/32.png) [@simoloca](https://discuss.pytorch.org/u/simoloca)\
**Post date:** [September 26, 2020, 4:41pm UTC](https://discuss.pytorch.org/t/errors-when-using-num-workers-0-in-dataloader/97564/6 "2020-09-26T16:41:27Z")

</div>

Ok, thanks you anyway!

---

<div class="post-metadata">

**Author:** ![kartikpodugu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kartikpodugu/32/12673_2.png) [@kartikpodugu](https://discuss.pytorch.org/u/kartikpodugu)\
**Post date:** [December 7, 2021, 11:44am UTC](https://discuss.pytorch.org/t/errors-when-using-num-workers-0-in-dataloader/97564/7 "2021-12-07T11:44:05Z")

</div>

Thanks for the solution. It worked for me.  
But, same question. How did it solve the problem?

---

<div class="post-metadata">

**Author:** ![algeriapy](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/34f0e0/32.png) [@algeriapy](https://discuss.pytorch.org/u/algeriapy)\
**Post date:** [January 29, 2024, 7:49pm UTC](https://discuss.pytorch.org/t/errors-when-using-num-workers-0-in-dataloader/97564/8 "2024-01-29T19:49:24Z")

</div>

Great !! 🥳 🥳 🥳  
if works, but the question how it works 😵‍💫 😵‍💫 😵‍💫 😵‍💫

Thanks for sharing it !!!