I have a large dataset. How can I make a selection from this set in order in steps? I use Dataloader, it allows you to do a bat by random sampling or in order. I want the batch to be configured in order, but in increments. For example, 1, 10, 20, 30, 40, 50. This is the first batch. Then, 2, 11, 21, 31, 41, 51 … So can I do this?
You could write your custom sampler and pass it to the DataLoader, which should return the indices as you need them.
Alternatively, you could reduce the length of your Dataset by the factor of your batch size, and create the batch in __getitem__ manually using batch_size=1 in your DataLoader.
While this approach could be simpler to implement, each batch will be created by a single worker, which might slowdown your code, if the bottleneck is in data loading.