Dataset.__getitem__() queries an external database every time when a sample is fetched. I didn’t expect it to work with
num_workers>=2, but at least with
num_workers=1, to let pytorch fetch the next batch while training the NN on the foregoing one (is that what
num_workers=1 does?). It fails even then with Segmentation Fault, so I’m using
Is there any trick that could help me? Like avoiding all calls to
__getitem__() (for e.g. doing data checks prior to training) and let torch alone call it during the training? Or maybe wrapping pieces in critical sections (no clue so far how to do it in python though).
I read DataLoader, when num_worker >0, there is bug but couldn’t find a workaround there. Used version: 1.0.0 py3.6_cuda9.0.176_cudnn7.4.1_1