How could I reset dataloader or count data batch with iter instead of epoch

The can only provide data batch of one epoch. How could I reset it before it accomplish one epoch so that it will not raise a stopIteration error unexpectedly.

If I understand you correctly, you want to infinitly loop over your dataloader until a breaking condiction is matched?

You could do something like this (assuming your Dataloader instance is stored in variable loader):

loader_iter = iter(loader)

while True:

        current_batch = next(loader_iter)
    except StopIteration:
        loader_iter = iter(loader)
        current_batch = next(loader_iter)

    ... # your following data processing steps

    if breaking_condition:

I tried this method, it worked well but I am afraid this might have some multi-process problems ?

  File "", line 16, in main
  File "/data/zhangzy/learn-projects/Segmentatron/SegNet/", line 90, in train
    im, label = next(trainiter)
  File "/home/zhangzy/.local/lib/python2.7/site-packages/torch/utils/data/", line
 280, in __next__
    idx, batch = self._get_batch()
  File "/home/zhangzy/.local/lib/python2.7/site-packages/torch/utils/data/", line
 259, in _get_batch
    return self.data_queue.get()
  File "/usr/lib/python2.7/multiprocessing/", line 378, in get
    return recv()
  File "/home/zhangzy/.local/lib/python2.7/site-packages/torch/multiprocessing/", line
 22, in recv
    return pickle.loads(buf)
  File "/usr/lib/python2.7/", line 1388, in loads
    return Unpickler(file).load()
  File "/usr/lib/python2.7/", line 864, in load
  File "/usr/lib/python2.7/", line 1139, in load_reduce
    value = func(*args)
  File "/home/zhangzy/.local/lib/python2.7/site-packages/torch/multiprocessing/",
 line 68, in rebuild_storage_fd
    fd = multiprocessing.reduction.rebuild_handle(df)
  File "/usr/lib/python2.7/multiprocessing/", line 157, in rebuild_handle
    new_handle = recv_handle(conn)
  File "/usr/lib/python2.7/multiprocessing/", line 83, in recv_handle
    return _multiprocessing.recvfd(conn.fileno())
OSError: [Errno 4] Interrupted system call
Exception NameError: "global name 'FileNotFoundError' is not defined" in <bound method _DataLo
aderIter.__del__ of < object at 0x7f15845d99d0>> ig

If you want to create an ‘infinite generator’ simply place it in a while loop:

def cycle(iterable):
    while True:
        for x in iterable:
            yield x

However there is a more elegant solution with a DataLoader. Simply enumerate over the dataloader when training for one epoch:

data_loader =
def train(epoch):
        for batch_idx, (data, target) in enumerate(data_loader):
            # Your code for training one epoch.

# Now perform training for however many epochs you want.
for epoch in range(num_epochs):

Internally every time you call enumerate(data_loader), it will call the object’s iter method which will create and return a new _DataLoaderIter object, which internally calls iter() on your batch sampler here.


How could I make training cease at certain iter number in this way ?


if you want to stop after a certain iterations, you could simply use epoch \times batch_idx as an iteration count right ? and then terminate the code once epoch \times batch_idx reaches a certain threshold.

1 Like