My Dataset.__getitem__() queries an external database every time when a sample is fetched. I didn’t expect it to work with num_workers>=2, but at least with num_workers=1, to let pytorch fetch the next batch while training the NN on the foregoing one (is that what num_workers=1 does?). It fails even then with Segmentation Fault, so I’m using num_workers=0 now.
Is there any trick that could help me? Like avoiding all calls to __getitem__() (for e.g. doing data checks prior to training) and let torch alone call it during the training? Or maybe wrapping pieces in critical sections (no clue so far how to do it in python though).
I read DataLoader, when num_worker >0, there is bug but couldn’t find a workaround there. Used version: 1.0.0 py3.6_cuda9.0.176_cudnn7.4.1_1