I’ve written a data loader for the cityscapes and kitti dataset, both stored on HDDs, all preprocessing is on torch-tensors (only center crop, hr and vr flips. All done through t[:, top:bot, left:right] slices or torch.flips(t, [])).
I’ve tried PIL, cv2, and skimage-io for reading images, but there’s no difference in i/o speed.
I can see the GPU starving (usage = 0%) in smi. which happens every n=num_workers iteration.
is this a disk bottleneck or am I missing some optimization?
Problems characteristics
visible in cases when:
- batch_size >= 4
- num_worker >= 4
- torch 1.0 stable
- machine: google cloud compute (4 core >= skylake, 16gb, 1x p100)