I’m using torch.utils.data.DataLoader
with a custom data.Dataset
for loading video batches.
Since video is quiet heavy, I’m experiencing some latency issues, especially with the first batches of every epoch. For example:
2020-09-14 00:16:42,229 Epoch 120/200 Iteration 0/322 Loss: 9.586519445292652e-05 Time: 22.14 seconds/iteration
2020-09-14 00:18:24,911 Epoch 120/200 Iteration 50/322 Loss: 0.00014619871217291802 Time: 2.447 seconds/iteration
2020-09-14 00:19:35,139 Epoch 120/200 Iteration 100/322 Loss: 9.589765249984339e-05 Time: 1.931 seconds/iteration
2020-09-14 00:21:04,289 Epoch 120/200 Iteration 150/322 Loss: 8.543662988813594e-05 Time: 1.882 seconds/iteration
2020-09-14 00:22:16,219 Epoch 120/200 Iteration 200/322 Loss: 9.707502613309771e-05 Time: 1.772 seconds/iteration
2020-09-14 00:23:53,514 Epoch 120/200 Iteration 250/322 Loss: 0.00016338478599209338 Time: 1.806 seconds/iteration
2020-09-14 00:25:03,165 Epoch 120/200 Iteration 300/322 Loss: 0.00013553885219153017 Time: 1.738 seconds/iteration
Notice how long are the first iterations of the training of this epoch, and it actually happens in every epoch. I assume it is because the iter method just resets the loader for each epoch, so I was wondering if there’s any solution for warming it up? I already maximized num_workers
by setting it to 16.