I have defined a custom dataset where getitem involves some heavy processing.
When running a dataloader i get the error
(not passed num_workers to the dl)
train_image, train_labels = next(iter(train_dl))
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/multiprocessing/queues.py", line 242, in _feed
send_bytes(obj)
File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/multiprocessing/queues.py", line 232, in _feed
close()
File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 177, in close
self._close()
File "/opt/conda/lib/python3.7/multiprocessing/connection.py", line 361, in _close
_close(self._handle)
OSError: [Errno 9] Bad file descriptor
However a value is assigned to train_images and train_labels.
I believe this error has to do with multithreading/forking and pipes in the dataloader implementation.
How can this be resolved?
Thanks