Custom Dataset error: unable to mmap memory: you tried to mmap 0GB

[chsasank (Sasank Chilamkurthy)]

Related discussions:

Mmap memory error when use multiple CPU on Azure

Hi all, I am using multiple CPUS to train my model on azure with MongoDB. It seems I need to open a connection to data in each of the threads. Then I got this error. Traceback (most recent call last): File "main.py", line 225, in <module> model.share_memory() File "/home/textiq/anaconda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 468, in share_memory return self._apply(lambda t: t.share_memory_()) File "/home/textiq/anaconda/lib/python3.6/site-packages/torch/nn…

Loading huge data functionality

This way is good for images but isn’t fit for text. In NLP, data is usually in one file with multiple lines instead of one image in one file. So how to customize Dataset accordingly?