Dataloader StopIteration takes a long time

1

After a dataloader has iterated all data, it can move on to the next iteration without reinitialising. I’ve seen code like this.

    dataloader = DataLoader()
    for epoch in range(10):
        for data, label in dataloader:
            train(data, label)

I observed that after each round of dataloader finished loading data, a new round took a very long time to start. So I wrote a program to verify this process.

    dataloader = getDataProvider()    # length = 100  batch_size = 20
    iter = dataloader.__iter__()
    for i in range(10):
        start = time.time()
        try:
            data, label = iter.next()
        except StopIteration:
            print('StopIteration')
            iter = data.__iter__()
            data, label = iter.next()
        end = time.time()
        print(i, '  takes time  ', end - start)

The cycle i=5 in which the StopIteration exception occurred took 27 seconds. All other cycles take no more than 1 second. Even reinitialising a dataloader takes less than 1 second.

I know that dataloader does some reset function after a StopIteration exception, but why is it more time consuming than reinitialising a dataloader? What is the correct way to reuse a dataloader?

2

I changed the parameter ‘num_workers’ of dataloader. Then I found some interesting thing. _SingleProcessDataLoaderIter doesn’t hava this problem. _MultiProcessingDataLoaderIter takes a long time when the StopIteration exception occurs.

3

You could try to use persistent_workers=True while creating the DataLoader to avoid shutting down and recreating the processes.

4

Thank you. It solves my problem.