Confusing Time delay between batches

nlp
1

I’m using torch text and there is a strange behavior which I’m unable to understand here. I’m training a CNN on SST-2 dataset. Training time for one epoch is ~5 seconds. However, if I iterate over just batches, time taken is ~100ms, and just predicting takes a total of ~300ms. I’m unable to understand where time delay is

import time
total = 0
base = time.time()
for b in loader.train_loader:
    now = time.time()
    res = model(b.text)
    tm = time.time() - now
    total += tm
print("Prediction total time", total)
print("Elapsed time", time.time() - base)

Output is

Prediction total time 0.36881256103515625
Elapsed time 5.046782493591309

For just iterating through batches

import time
total = 0
base = time.time()
for b in loader.train_loader:
    len(b.text)
    tm = time.time() - now
    total += tm
print("Elapsed time", time.time() - base)

output is

Elapsed time 0.0856020450592041

Can anyone explain what I’m missing here? Thanks

2

Hi,
It seems that the model itself doesn’t take as much time to process the data as much as the dataloader takes to load it [ Note that you are timing the ‘prediction time’ after the dataloader has loaded and as such this is not reflected in the ‘prediction time’ ]. It might be due to the fact that your data is huge and the model is pretty lightweight. Also, your methods of reading data using the dataloader might be inefficient which is causing the delay. Pre-processing the data in the dataloader can also cause delays like these.