Questions about the data loader iter usage

oasjd7 · November 29, 2018, 12:24pm

I have to use both MNIST and SVHN dataset.
but length of MNIST train loader is 600 and SVHN train loader is 733. It’s different…
and I think I should reset the ‘data_iter’ on each dataset but I don’t know this usage is right.

# MNIST
dataset = dsets.MNIST(root='./data',
                      train=True,
                      transform=transforms.ToTensor(),
                      download=True)

mnist_data_loader = torch.utils.data.DataLoader(dataset=dataset,
                                          batch_size=100,
                                          shuffle=True)

# SVHN
svhn_transform = transforms.Compose([transforms.Grayscale(), transforms.Resize(28), transforms.ToTensor()])
dataset = dsets.SVHN(root='./data/',
                     split='train',
                     transform=svhn_transform,
                     download=True)
svhn_data_loader = torch.utils.data.DataLoader(dataset=dataset,
                                          batch_size=100,
                                          shuffle=True)

mnist_iter_per_epoch = len(mnist_train_loader) # length of this loader : 600
svhn_iter_per_epoch = len(svhn_train_loader) # length of this loader : 733

for epoch in range(100):
   # Reset the data_iter
   if (epoch+1) % mnist_iter_per_epoch ==0:
         mnist_iter = iter(mnist_train_loader)
   if (epoch+1) % svhn_iter_per_epoch ==0:
         svhn_iter = iter(svhn_train_loader)
    ...

ptrblck · November 29, 2018, 2:33pm

The length of a DataLoader returns the number of batches.
I’m not sure what’s your current use case, as you are somehow using the number of batches and the current epoch to reset the iterator.
Could you explain your use case a bit?

oasjd7 · November 30, 2018, 3:34am

github.com

bfarzin/pytorch_aae/blob/master/main_aae.py

import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision.datasets as dsets
import torchvision.transforms as transforms
from torch.autograd import Variable
from logger import Logger


# MNIST Dataset 
dataset = dsets.MNIST(root='./data', 
                      train=True, 
                      transform=transforms.ToTensor(),  
                      download=True)

# Data Loader (Input Pipeline)
data_loader = torch.utils.data.DataLoader(dataset=dataset, 
                                          batch_size=100, 
                                          shuffle=True)

This file has been truncated. show original

In this code they use iter. and I want to use another dataset at the same time for my purpose.

When the length of the dataloder is the different(mnist loader:600, schn loader:733), if I reset the iter with mnist loader I’m afraid that I cannot use 133 data from svhn. and not sure the one iter_reset can do the job when use 2 dataset.

vmirly1 · November 30, 2018, 4:57am

How about using a try … except block for each of the two loaders so that when each one of them runs out, it can get reset independent of the other one. So here I’m giving an example with tow loaders loader_A and loader_B:

# Loading a batch from A:
try:
     x_a, y_a = next(data_iter_A)
except:
     # loader A needs to reset
     data_iter_A = iter(loader_A)
     x_a, y_a = next(data_iter_A)

# Loading a batch from B:
try:
     x_b, y_b = next(data_iter_B)
except:
     # loader B needs to reset
     data_iter_A = iter(loader_B)
     x_b, y_b = next(data_iter_B)

oasjd7 · December 1, 2018, 4:55am

Thanks for the reply.

A_train_loader have 600 length, and B_train_loader have 733.
If I use your recommend code, can I use all data in both A and B loader?

vmirly1 · December 1, 2018, 3:59pm

Yes, it will cycle through A first, and then resets A while still loading from B to finish batches in B.