Training with a random fraction data from data loader every epoch

1

Hi,

I have a very large dataset to train my model. I would like to get n different random batches from the DataLoader at every different epoch. How can I do it?

Thanks,

2

You could create an iterator via loader_iter = iter(loader) and get the next batch via batch = next(iter). If you’ve set shuffle=True in the DataLoader, these batches will contain random samples.

3

Hello i cannot quite understand your answer could you please expand on it

4

Could you describe what exactly is unclear in my post and where you are currently stuck?

5

Well honestly speaking am quite new to pytorch.
So I have a very huge dataset that it is taking a lot of time during the training process. I just need the dataloader to use a fraction of the dataset for every iteration.
Please help make your answer friendly for a newbie like me

6

If you want to load a specific subset of the Dataset, use torch.utils.data.Subset and pass your Dataset object as well as the desired indices to it.
My code snippet is manually creating an iterator and is getting a single batch via the next call, which doesn’t seem to match your use case.

1 Like