How to dynamically adjust dataset within a epoch

1

Hi There,

I want to dynamically update the dataset within a epoch. That is something like:

dataset = ImageFolder()
dataloader = DataLoader()

for e in range(epoches):
    for batch in dataloader:
       # Do some job
      ....
       # Update the dataset

Thank you so much!

2

There are likely many ways of doing this depending on what “update the dataset” actually means. If you merely need to do something like tweak transformations, then this might be doable with some global state that is used by your transform/preprocessing functions. If it means adding or removing samples, then this becomes substantially trickier as it involves either changing the data loading iterator (torch.utils.data.dataloader — PyTorch 1.8.1 documentation) or building a new dataset and dataloader on the fly using the remaining data.

1 Like