# Resume iterating dataloader from checkpoint batch\_idx

**URL:** <https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683>\
**Category:** Uncategorized\
**Created:** [November 12, 2019, 2:07am UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683 "2019-11-12T02:07:55Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![SU801T](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/32.png) [@SU801T](https://discuss.pytorch.org/u/SU801T)\
**Post date:** [November 12, 2019, 2:07am UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/1 "2019-11-12T02:07:55Z")

</div>

Hi,

I was wondering whether it is possible to resume iterating through a dataloader from a checkpoint.

For example:

```auto
    dataloaders_dict = {phase: torch.utils.data.DataLoader(datasets_dict[phase], batch_size=args.batch_size, num_workers=args.num_workers, shuffle=False) for phase in ['train']} # make sure shuffling is false incase you restart

if os.path.isdir(args.batchidx_checkpoint):
            checkpoint = torch.load(args.batchidx_checkpoint + 'batches.pt')
            batch_idx = checkpoint['batch_idx'] + 1
     
        else:
            os.makedirs(args.batchidx_checkpoint)
            batch_idx = 0

for batch_idx, (inputs, labels) in enumerate(dataloaders_dict['train']):
            #print(batch_idx)
            # do some feature extraction and save feature arrays in hdf5 file  

```

I have seen some examples mentioning random sampler, but was wondering if it is really necessary…  
I also have about 5 million images to go through, so it may be a large dataset to iterate through. My end goal is to save features for each image using a pretrained network, thus no training is really required. I just want to resume feature extraction from a checkpoint batch id…

Cheers,

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [November 12, 2019, 4:48am UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/2 "2019-11-12T04:48:27Z")

</div>

Hi,

I don’t think you can do that using the vanilla modules.  
But it should be fairly easy to make your own sampler inspired by the [`SequentialSampler()`](https://github.com/pytorch/pytorch/blob/8db06732bfe4f3aa817c984f85242c94b78c00f4/torch/utils/data/sampler.py#L51-L65) that is 15 lines of code to add a new argument to pass in a start point with `return iter(range(self.start_idx, len(self.data_source)))`.

---

<div class="post-metadata">

**Author:** ![SU801T](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/32.png) [@SU801T](https://discuss.pytorch.org/u/SU801T)\
**Post date:** [November 12, 2019, 6:25pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/3 "2019-11-12T18:25:07Z")

</div>

Hi,

Thanks for the reply. I created a new SequentialSampler class in my script:

```auto
class SequentialSampler2():
    r"""Samples elements sequentially, always in the same order.
    Arguments:
        data_source (Dataset): dataset to sample from
    """

    def __init__ (self, data_source):
        self.data_source = data_source

    def __iter__ (self):
        return iter(range(self.start_idx, len(self.data_source)))

    def __len__ (self):
        return len(self.data_source)

```

and try to pass it to sampler in my dataloader, like this:

```auto
datasets_dict = {phase: Patches(root= args.root_path + phase, phase = phase) for phase in ['train']}
sampler = {phase: SequentialSampler2(len(datasets_dict[phase])) for phase in ['train']}
dataloaders_dict = {phase: torch.utils.data.DataLoader(datasets_dict[phase], batch_size=args.batch_size, sampler=sampler, num_workers=args.num_workers, shuffle=False) for phase in ['train']} # make sure shuffling is false incase you restart

resnet50 = models.resnet50(pretrained=True)
for param in resnet50.parameters():
    param.requires_grad = False

# check batch checkpoint 
if os.path.isdir(args.batchidx_checkpoint):
            checkpoint = torch.load(args.batchidx_checkpoint + 'batches.pt')
            batch_idx = checkpoint['batch_idx'] + 1
   
else:
        os.makedirs(args.batchidx_checkpoint)
        batch_idx = 0

# do some other stuff 

for batch_idx, (inputs40x, inputs20x, inputs5x, paths40x, paths20x, paths5x, labels) in enumerate(dataloaders_dict[phase]):

            inputs40x = inputs40x.to(device)
            inputs20x = inputs20x.to(device)
            inputs5x = inputs5x.to(device)

            labels = labels.to(device)

```

however, appear to get an error:

```auto
ValueError: sampler should be an instance of torch.utils.data.Sampler, but got sampler={'train': < __main__.SequentialSampler2 object at 0x128b2d748>}

```

Perhaps I’ve misunderstood how to feed in a sampler to the dataloader. Also I’m not sure whether this is the correct approach to consider when restarting a certain batch\_idx in the for loop.

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [November 12, 2019, 8:12pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/4 "2019-11-12T20:12:18Z")

</div>

It’s just that your sampler should be a subclass of the original `torch.utils.data.Sampler` as the error states.  
You can fix that by setting `class SequentialSampler2(torch.utils.data.Sampler):` to inherit from it. and add `super(). __init__ ()` in the first line of your init function to make sure to initialize the parent class.

---

<div class="post-metadata">

**Author:** ![SU801T](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/94ad74/32.png) [@SU801T](https://discuss.pytorch.org/u/SU801T)\
**Post date:** [November 12, 2019, 9:56pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/5 "2019-11-12T21:56:28Z")

</div>

ahh, ok thank you! 🙂

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [May 2, 2020, 3:37pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/6 "2020-05-02T15:37:34Z")

</div>

Why does pytorch not have this working out of the box? Doesn’t this make it much harder to have reproducible ML by not having this easily?

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [May 4, 2020, 2:33pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/7 "2020-05-04T14:33:52Z")

</div>

Hi,

I’m sure this could be added but there was no explicit feature request for it.  
Could you clarify how this is particularly important for reproducible ML? You often want to start your dataset from the middle?

---

<div class="post-metadata">

**Author:** ![pinocchio](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/32/22093_2.png) [@pinocchio](https://discuss.pytorch.org/u/pinocchio)\
**Post date:** [May 8, 2020, 3:35pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/8 "2020-05-08T15:35:19Z")

</div>

This is crucial when saving checkpoints (because we have a cluster that only allows to run 1day jobs and then I have to re-start it another day from the exact place).

Reproducibility in Deep Learning is a very big issue and this doesn’t make it better.

I’d make a feature request.

---

<div class="post-metadata">

**Author:** ![albanD](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/32/215_2.png) [@albanD](https://discuss.pytorch.org/u/albanD)\
**Post date:** [May 8, 2020, 4:13pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/9 "2020-05-08T16:13:00Z")

</div>

I see the point about resuming training.  
But do you actually save all the random state when you checkpoint? The sampler won’t continue sampling exactly the same thing otherwise.

---

<div class="post-metadata">

**Author:** ![gabrielwong1991](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gabrielwong1991/32/30100_2.png) [@gabrielwong1991](https://discuss.pytorch.org/u/gabrielwong1991)\
**Post date:** [August 1, 2021, 3:34pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/10 "2021-08-01T15:34:41Z")

</div>

Hi all, recently I also have this problem and my Sampler is using `WeightedRandomSampler`. I tried to added @albanD 's solution but unable to do so with my custom sampler.

```auto
class WeightedRandomSampler2(torch.utils.data.Sampler):
    r"""Samples elements from ``[0,..,len(weights)-1]`` with given probabilities (weights).
    Args:
        weights (sequence) : a sequence of weights, not necessary summing up to one
        num_samples (int): number of samples to draw
        replacement (bool): if ``True``, samples are drawn with replacement.
            If not, they are drawn without replacement, which means that when a
            sample index is drawn for a row, it cannot be drawn again for that row.
    Example:
        >>> list(WeightedRandomSampler([0.1, 0.9, 0.4, 0.7, 3.0, 0.6], 5, replacement=True))
        [0, 0, 0, 1, 0]
        >>> list(WeightedRandomSampler([0.9, 0.4, 0.05, 0.2, 0.3, 0.1], 5, replacement=False))
        [0, 1, 4, 3, 2]
    """

    def __init__ (self, weights, num_samples, replacement=True):

      super(). __init__ ()

      if not isinstance(num_samples, _int_classes) or isinstance(num_samples, bool) or \
              num_samples <= 0:
          raise ValueError("num_samples should be a positive integer "
                           "value, but got num_samples={}".format(num_samples))
      if not isinstance(replacement, bool):
          raise ValueError("replacement should be a boolean value, but got "
                           "replacement={}".format(replacement))
      self.weights = torch.as_tensor(weights, dtype=torch.double)
      self.num_samples = num_samples
      self.replacement = replacement

    def __iter__ (self):
      return iter(self.start_idx, torch.multinomial(self.weights, self.num_samples, self.replacement).tolist())

    def __len__ (self):
      return self.num_samples

```

That is by adding `self.start_idx` in ` __iter__ ()` I received the error when running it in the script:

```auto
data_sampler = WeightedRandomSampler2(sample_weights, num_samples=len(sample_weights), replacement=True)

```

```auto
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-14-b420c064a884> in <module>()
----> 1 data_sampler = WeightedRandomSampler2(sample_weights, num_samples=len(sample_weights), replacement=True)

<ipython-input-10-3c0a1378cd2d> in __init__ (self, weights, num_samples, replacement)
     20 def __init__ (self, weights, num_samples, replacement=True):
     21 
---> 22 super(). __init__ ()
     23 
     24 if not isinstance(num_samples, _int_classes) or isinstance(num_samples, bool) or num_samples <= 0:

TypeError: __init__ () missing 1 required positional argument: 'data_source'

```

Any idea?

---

<div class="post-metadata">

**Author:** ![Jack\_Hu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jack_hu/32/62364_2.png) [@Jack\_Hu](https://discuss.pytorch.org/u/Jack_Hu)\
**Post date:** [August 24, 2023, 9:06am UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/11 "2023-08-24T09:06:39Z")

</div>

Saving all the random state is essential for reproducing and debugging. For reproducing, the author offere the random seed and code, and we readers use the same settings to reproduce the results. If resuming happens in author’s training or readers’ training process, the results are different. You might expect the algorithm to be stable for different random states, but this setting is just for the guarantee of reproducing the results. This could offer us great convenience, and not worrying about the sudden code failure due to reasons like “no space on hard disk”. Moreover, for debugging, it is surely useful when the bug occurs with some certain data after many iterations like within batch 100000. When I save the checkpoint of batch 99999, then I resume training for debug several times, the dataloader is loading the batch 0 for me! This is so inconvenient! So I strongly suggest this feature, and it should be a optional setting, turn on for debugging and reproducing, turn off for simple training. If more questions exist, I’d like to continue discussing.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [August 24, 2023, 1:51pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/12 "2023-08-24T13:51:12Z")

</div>

You’ve mentioned some good ideas! Would you be interested in implementing these debugging utils.?

---

<div class="post-metadata">

**Author:** ![Jack\_Hu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jack_hu/32/62364_2.png) [@Jack\_Hu](https://discuss.pytorch.org/u/Jack_Hu)\
**Post date:** [August 24, 2023, 4:52pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/13 "2023-08-24T16:52:26Z")

</div>

I am surely interested. As a pytorch heavily user, I’m quite familiar with the APIs. As for implementing, maybe I need some time to first read the source code of the dataloader and the dataset. I am currently reading and working on it, but I’m quite a rookie in this. If any plans or hints exist for implementing this, please help me with it.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [August 24, 2023, 5:39pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/14 "2023-08-24T17:39:27Z")

</div>

You could start by reading through the [Contribution Guide](https://pytorch.org/docs/stable/community/contribution_guide.html) and start with a feature request discussing your ideas. After discussion the features with the code owners you could then start implementing these once you are aligned.

---

<div class="post-metadata">

**Author:** ![Jack\_Hu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jack_hu/32/62364_2.png) [@Jack\_Hu](https://discuss.pytorch.org/u/Jack_Hu)\
**Post date:** [August 25, 2023, 1:36am UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/15 "2023-08-25T01:36:00Z")

</div>

Thanks a lot! It’s my first time trying to contribute to pytorch, this is really helpful to me. I’ll start working on this.

---

<div class="post-metadata">

**Author:** ![xgbj](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xgbj/32/67179_2.png) [@xgbj](https://discuss.pytorch.org/u/xgbj)\
**Post date:** [March 18, 2024, 3:23pm UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/16 "2024-03-18T15:23:56Z")

</div>

Hi, is there any updates on this topic, I’m facing the same needs too.

---

<div class="post-metadata">

**Author:** ![xgbj](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xgbj/32/67179_2.png) [@xgbj](https://discuss.pytorch.org/u/xgbj)\
**Post date:** [March 19, 2024, 2:52am UTC](https://discuss.pytorch.org/t/resume-iterating-dataloader-from-checkpoint-batch-idx/60683/17 "2024-03-19T02:52:07Z")

</div>

Hello, I found a similar implementation online that seems to solve our problem. [vissl/vissl/data/data\_helper.py at main · facebookresearch/vissl · GitHub](https://github.com/facebookresearch/vissl/blob/main/vissl/data/data_helper.py#L93)  
it looks like we can set\_start\_iter in StatefulDistributedSampler to implement the stateful resume requirement we want.