Why batch loading time varies for different batches (Image Loading)?

[ptrblck]

@ammary-mo

But are you saying that my data loading is slow compared with training loop time

Yes, it seems that the data loading time is large compared to the actual model training time, which would explain the peaks in the data loading time.

@ammary-mo

or you mean that I have an issue with MyDataset class?

There might be a potential improvement, if replacing:

torch.tensor(self.datasets[index][1], dtype=torch.long)

which will trigger a copy with:

torch.from_numpy(self.datasets[index][1]astype(np.int64))

which would reuse the underlying data (assuming self.datasets returns a numpy array).

@ammary-mo

What I understand from the discussion, that in my case, there is no way to avoid this other than increasing the number of workers to the limit that makes their loading time hidden compared with the training loop time. Please correct my understanding.

Yes, that’s correct and you would have to either speed up the data loading (or increase the workload for the model training).
Note that, the smaller the GPU workload is, the more likely you’ll hit a data loading bottleneck.
In the extreme case that your model immediately trains an iteration (just remove the model training), you would have to make sure that the data loading is fast enough to load and process the next batch while Python “executes the loop” (which would be fast as there is no real workload).