I am using pytorch data loader and runs into some strange error as below, we generate training data in each hour. and I run this experiment to just read training data csv file and parse line by line, no real training stuff is going. we define the dataloader each item as a file in bunch of date/hour parsed folder. and run multi worker data loader
I checked it should not be __get_item index out of boundary, is it because some training file we loaded is too large? it is not always repeatable error, so I am not sure it may be related to some training data? we constantly generate training data every hour, so any hint?
Traceback (most recent call last):
File "/home/miniconda/lib/python3.6/multiprocessing/queues.py", line 240, in _feed
send_bytes(obj)
File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/miniconda/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647