# Data loader crashes during training. Something to do with multiprocessing in docker

**URL:** <https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379>\
**Category:** Uncategorized\
**Created:** [June 27, 2017, 11:28pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379 "2017-06-27T23:28:14Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![SiddGururani](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siddgururani/32/9142_2.png) [@SiddGururani](https://discuss.pytorch.org/u/SiddGururani)\
**Post date:** [June 27, 2017, 11:28pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/1 "2017-06-27T23:28:14Z")

</div>

Hi, I’m training a variant of Baidu’s deepspeech model using the code from [this repository](https://github.com/SeanNaren/deepspeech.pytorch). The dataset is around 1000 hours of audio.

While running the [train.py](http://train.py) script, I’m running into this error:

```
Traceback (most recent call last):
  File "train.py", line 411, in <module>
    main()
  File "train.py", line 219, in main
    for i, (data) in enumerate(train_loader, start=start_iter):
  File "/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 195, in __next__
    idx, batch = self.data_queue.get()
  File "/opt/conda/envs/pytorch-py35/lib/python3.5/multiprocessing/queues.py", line 345, in get
    return ForkingPickler.loads(res)
  File "/opt/conda/envs/pytorch-py35/lib/python3.5/site-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
    fd = df.detach()
  File "/opt/conda/envs/pytorch-py35/lib/python3.5/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/opt/conda/envs/pytorch-py35/lib/python3.5/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/opt/conda/envs/pytorch-py35/lib/python3.5/multiprocessing/connection.py", line 487, in Client
    c = SocketClient(address)
  File "/opt/conda/envs/pytorch-py35/lib/python3.5/multiprocessing/connection.py", line 614, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused

```

I’m using nvidia-docker for my environment and I’ve set the --ipc=host flag which is recommended for having shared memory. Anyone have any idea why I’m running into this?  
I’ve not faced this issue in the past when I trained on a local installation of Pytorch (without docker) so I could try to run it again outside of docker but I’d like to keep my environment as a docker image.

---

<div class="post-metadata">

**Author:** ![mromaniuk](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ccd318/32.png) [@mromaniuk](https://discuss.pytorch.org/u/mromaniuk)\
**Post date:** [August 16, 2017, 10:35am UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/2 "2017-08-16T10:35:27Z")

</div>

I’m also having this problem (but with Python 3.6 on Anaconda). It only happens when running on Docker (with `--ipc=host`).

In my case it seems to be caused by a problem in one of the worker processes:

```auto
Process Process-4:
Traceback (most recent call last):
  File "/miniconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/miniconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/miniconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 44, in _worker_loop
    data_queue.put((idx, samples))
  File "/miniconda3/lib/python3.6/multiprocessing/queues.py", line 349, in put
    obj = _ForkingPickler.dumps(obj)
  File "/miniconda3/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
  File "/miniconda3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 113, in reduce_storage
    fd, size = storage._share_fd_()
RuntimeError: unable to write to file </torch_31_2143820281> at /opt/conda/conda-bld/pytorch_1502009910772/work/torch/lib/TH/THAllocator.c:271

```

This then seems to cause the error in the main process:

```auto
File "/miniconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 195, in __next__
    idx, batch = self.data_queue.get()
  File "/miniconda3/lib/python3.6/multiprocessing/queues.py", line 345, in get
    return _ForkingPickler.loads(res)
  File "/miniconda3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
    fd = df.detach()
  File "/miniconda3/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/miniconda3/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/miniconda3/lib/python3.6/multiprocessing/connection.py", line 487, in Client
    c = SocketClient(address)
  File "/miniconda3/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused

```

---

<div class="post-metadata">

**Author:** ![alxndrkalinin](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alxndrkalinin/32/1767_2.png) [@alxndrkalinin](https://discuss.pytorch.org/u/alxndrkalinin)\
**Post date:** [September 26, 2017, 5:26pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/3 "2017-09-26T17:26:48Z")

</div>

Did anyone solve this? I have the same problem running model in nvidia-docker.

---

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [September 27, 2017, 7:05pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/4 "2017-09-27T19:05:16Z")

</div>

we’re not able to get a reliable repro of this problem to investigate and fix this.  
If anyone can give a docker repro of the problem we can look into this further.

---

<div class="post-metadata">

**Author:** ![Elias\_Vansteenkiste](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elias_vansteenkiste/32/836_2.png) [@Elias\_Vansteenkiste](https://discuss.pytorch.org/u/Elias_Vansteenkiste)\
**Post date:** [October 12, 2017, 3:50pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/5 "2017-10-12T15:50:03Z")

</div>

I also have the same problem.

```
Traceback (most recent call last):
  File "train.py", line 20, in <module>
    for i, data in enumerate(dataset):
  File "/home/elias/.local/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 195, in __next__
    idx, batch = self.data_queue.get()
  File "/usr/lib/python3.5/multiprocessing/queues.py", line 345, in get
    return ForkingPickler.loads(res)
  File "/home/elias/.local/lib/python3.5/site-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/lib/python3.5/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/usr/lib/python3.5/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 487, in Client
    c = SocketClient(address)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 614, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused

```

I don’t use conda or miniconda, maybe that rules something out?

---

<div class="post-metadata">

**Author:** ![Elias\_Vansteenkiste](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elias_vansteenkiste/32/836_2.png) [@Elias\_Vansteenkiste](https://discuss.pytorch.org/u/Elias_Vansteenkiste)\
**Post date:** [October 12, 2017, 3:53pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/6 "2017-10-12T15:53:39Z")

</div>

Everything works fine if I set num\_workers to 0.

```
self.dataloader = torch.utils.data.DataLoader(
            self.dataset,
            batch_size=opt.batchSize,
            shuffle=not opt.serial_batches,
            num_workers=int(opt.nThreads))

```

obviously

---

<div class="post-metadata">

**Author:** ![FuriouslyCurious](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/furiouslycurious/32/294_2.png) [@FuriouslyCurious](https://discuss.pytorch.org/u/FuriouslyCurious)\
**Post date:** [October 12, 2017, 9:35pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/7 "2017-10-12T21:35:51Z")

</div>

Looks like you are running out of disk space when loading and preprocessing with data loader.

---

<div class="post-metadata">

**Author:** ![Elias\_Vansteenkiste](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elias_vansteenkiste/32/836_2.png) [@Elias\_Vansteenkiste](https://discuss.pytorch.org/u/Elias_Vansteenkiste)\
**Post date:** [October 13, 2017, 7:57am UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/8 "2017-10-13T07:57:02Z")

</div>

No that is not the case. I monitored the work memory usage and the disk space usage and for both I have more than 50% free space.

---

<div class="post-metadata">

**Author:** ![Elias\_Vansteenkiste](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elias_vansteenkiste/32/836_2.png) [@Elias\_Vansteenkiste](https://discuss.pytorch.org/u/Elias_Vansteenkiste)\
**Post date:** [October 13, 2017, 7:59am UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/9 "2017-10-13T07:59:08Z")

</div>

I came up on a related problem.  
If I don’t set the number of threads then sometimes the trainer hangs and I have to kill it with ctrl-c.

```
^CTraceback (most recent call last):
  File "train.py", line 20, in <module>
    for i, data in enumerate(dataset):
  File "/home/elias/.local/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 195, in __next__
    idx, batch = self.data_queue.get()
  File "/usr/lib/python3.5/multiprocessing/queues.py", line 343, in get
    res = self._reader.recv_bytes()
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
```

---

<div class="post-metadata">

**Author:** ![Zhixiang\_Wang](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/8dc957/32.png) [@Zhixiang\_Wang](https://discuss.pytorch.org/u/Zhixiang_Wang)\
**Post date:** [October 23, 2017, 1:05pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/10 "2017-10-23T13:05:48Z")

</div>

I am now facing the same issue.

---

<div class="post-metadata">

**Author:** ![Zhixiang\_Wang](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/8dc957/32.png) [@Zhixiang\_Wang](https://discuss.pytorch.org/u/Zhixiang_Wang)\
**Post date:** [October 24, 2017, 10:35am UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/11 "2017-10-24T10:35:27Z")

</div>

I solved this problem by increasing the size of shared memory by using `--shm-size`, i.e.,

```bash
nvidia-docker run -it --shm-size 8G --name xxxx xxxx

```

This issue dues to that the size of shared memory is not enough.

---

<div class="post-metadata">

**Author:** ![SpandanMadan](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spandanmadan/32/1371_2.png) [@SpandanMadan](https://discuss.pytorch.org/u/SpandanMadan)\
**Post date:** [October 30, 2017, 6:57am UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/12 "2017-10-30T06:57:13Z")

</div>

I am getting this same error without Docker. So there’s definitely something more to it! Did anyone find a fix?

---

<div class="post-metadata">

**Author:** ![thnkim](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thnkim/32/2808_2.png) [@thnkim](https://discuss.pytorch.org/u/thnkim)\
**Post date:** [December 18, 2017, 1:01pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/13 "2017-12-18T13:01:22Z")

</div>

I have the same issue. Exactly the same.  
I ran on Ubuntu 16.04, two 1080 Ti. num\_worker was 8, batch size was 96.

```auto
  File "train_cs.py", line 135, in <module>
    for i, data in enumerate(data_loader):
  File "/home/sipark/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 204, in __next__
    idx, batch = self.data_queue.get()
  File "/home/sipark/anaconda3/lib/python3.6/multiprocessing/queues.py", line 345, in get
    return _ForkingPickler.loads(res)
  File "/home/sipark/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
    fd = df.detach()
  File "/home/sipark/anaconda3/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/home/sipark/anaconda3/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/home/sipark/anaconda3/lib/python3.6/multiprocessing/connection.py", line 487, in Client
    c = SocketClient(address)
  File "/home/sipark/anaconda3/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused

```

---

<div class="post-metadata">

**Author:** ![tal](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/54ee81/32.png) [@tal](https://discuss.pytorch.org/u/tal)\
**Post date:** [January 12, 2018, 3:28pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/14 "2018-01-12T15:28:11Z")

</div>

Same here with python 2.7 and pytorch 0.2. Is it happening to anyone also on pytorch 0.3 or should I upgrade to try to solve this?

```
   batch = data_iter.next()
  File "/home/tals/env1/local/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 195, in __next__
idx, batch = self.data_queue.get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 378, in get
return recv()
  File "/home/tals/env1/local/lib/python2.7/site-packages/torch/multiprocessing/queue.py", line 22, in recv
return pickle.loads(buf)
  File "/usr/lib/python2.7/pickle.py", line 1388, in loads
return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 864, in load
dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1139, in load_reduce
value = func(*args)
  File "/home/tals/env1/local/lib/python2.7/site-packages/torch/multiprocessing/reductions.py", line 68, in rebuild_storage_fd
fd = multiprocessing.reduction.rebuild_handle(df)
  File "/usr/lib/python2.7/multiprocessing/reduction.py", line 155, in rebuild_handle
conn = Client(address, authkey=current_process().authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client
c = SocketClient(address)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 308, in SocketClient
s.connect(address)
  File "/usr/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 111] Connection refused
```

---

<div class="post-metadata">

**Author:** ![shiretzet](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/bbe5ce/32.png) [@shiretzet](https://discuss.pytorch.org/u/shiretzet)\
**Post date:** [February 5, 2018, 7:58pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/15 "2018-02-05T19:58:40Z")

</div>

Same for me 😕

---

<div class="post-metadata">

**Author:** ![tal](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/54ee81/32.png) [@tal](https://discuss.pytorch.org/u/tal)\
**Post date:** [February 5, 2018, 8:27pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/16 "2018-02-05T20:27:57Z")

</div>

Increasing the shared memory size solved this for me. It is used to share data between processes  
Try to track your ram during training and make sure that there is no memory leak

---

<div class="post-metadata">

**Author:** ![vincentgc](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/7ea924/32.png) [@vincentgc](https://discuss.pytorch.org/u/vincentgc)\
**Post date:** [April 19, 2018, 1:52pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/17 "2018-04-19T13:52:39Z")

</div>

I solved this problem by changing the num\_workers from 4 to 8. But I don`t know why.

---

<div class="post-metadata">

**Author:** ![JC-S](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/b19c9b/32.png) [@JC-S](https://discuss.pytorch.org/u/JC-S)\
**Post date:** [May 23, 2018, 5:11pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/18 "2018-05-23T17:11:34Z")

</div>

I had run into the same problem with CUDA 9.0 pytorch 0.3.1.  
I tried to change the num\_workers but it didn’t work.  
Oddly enough, it is fixed after I doubled the batch\_size.

---

<div class="post-metadata">

**Author:** ![luonango](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/f19dbf/32.png) [@luonango](https://discuss.pytorch.org/u/luonango)\
**Post date:** [June 25, 2018, 4:48am UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/19 "2018-06-25T04:48:32Z")

</div>

Increasing the size of shared memory may solve the problem. If not, try tu set the num\_workers=0.  
Using the command “df -h /dev/shm” can view the state of shared memory.

---

<div class="post-metadata">

**Author:** ![Ran](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ran/32/7989_2.png) [@Ran](https://discuss.pytorch.org/u/Ran)\
**Post date:** [August 7, 2018, 12:01pm UTC](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379/20 "2018-08-07T12:01:29Z")

</div>

I had the same error without docker, it appears without rhyme or reason！ I changed the batch\_size or num\_workers, but they can’t help me? Can anyone share your solution?  
The error is as follows:

```auto
Exception socket.error: error(111, 'Connection confused') in <bound method DataLoaderIter. __del__ of <torch.utils.data.dataloader.DataLoaderIter object at 0x7fb041d34690>> ignored

```

[Next page](https://discuss.pytorch.org/t/data-loader-crashes-during-training-something-to-do-with-multiprocessing-in-docker/4379.md?page=2)