How to loop through two data sets at once with iterators in Pytorch without getting a StopIteration error?

1

is the following the recommended way to do it in pytorch:

for epoch in range(nb_epochs):
    for train_data, test_data in zip(trainloader, testloader):
          #do train

is this the pytorch way to do it? There isn’t any inefficiencies or any subtle things I should worry about?

2

depending on how many elements large trainloader and testloader are, you could use the itertools zip: https://docs.python.org/2/library/itertools.html#itertools.izip

1 Like
3

I want to track the train and test error at the end of each epoch.

4

what would be the difference of using izip vs zip?

5

If you’re using Python 3, you should use zip. zip returns an iterator.

If you’re using Python 2, you can get some memory gains by using izip over zip. This is because in Python 2 zip returns a list instead of an iterator, while izip returns an iterator. If you’re iterating over very long lists, the lists will take up memory.

6

but in the context of DL, zip truncates my training to the size of the test set cuz test sets are usually smaller. I am sure someone has dealt with this before me…

7

I’m not sure what you’re looking for but Itertools has a zip_longest method that might be helpful: https://docs.python.org/3.0/library/itertools.html