# Does a dataloader change random state even when shuffle argument is False?

**URL:** <https://discuss.pytorch.org/t/does-a-dataloader-change-random-state-even-when-shuffle-argument-is-false/92569>\
**Category:** Uncategorized\
**Created:** [August 12, 2020, 2:24pm UTC](https://discuss.pytorch.org/t/does-a-dataloader-change-random-state-even-when-shuffle-argument-is-false/92569 "2020-08-12T14:24:33Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![jmaronas](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jmaronas/32/2816_2.png) [@jmaronas](https://discuss.pytorch.org/u/jmaronas)\
**Post date:** [August 12, 2020, 2:24pm UTC](https://discuss.pytorch.org/t/does-a-dataloader-change-random-state-even-when-shuffle-argument-is-false/92569/1 "2020-08-12T14:24:33Z")

</div>

Hello.

I am facing an unexpected behaviour in my dataloading scheme. I have two dataloaders, a `train_dl` and a `test_dl`. The `train_dl` provides batches of data with the argument `shuffle=True` and the `test_dl` provide batches with the argument `shuffle=False`.

I evaluate my test metrics each N epochs, i.e each N epochs I loop over `test_dl` dataset. I have realized that if the value of N changes, then the shuffled batches provided by the `train_dl` after evaluating the test metrics differ. In other words, suppose my training points are labelled: a,b,c,d,e,f and my batch size is 2. Then if I seed everything correctly I observe the following pattern always:

Epoch 1:  
Batch 1: [a,b]  
Batch 2: [e,d]  
Batch 3: [c,f]

Epoch 2:  
Batch 1: [a,f]  
Batch 2: [b,d]  
Batch 3: [e,c]

However, if in between epoch 1 and epoch 2 I loop over my `test_dl` then the provided batches at epoch 2 differ or in other words it seems that even `shuffle = False` for the `test_dl`, the random state seems to change.

Is this true?

Thanks

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [August 15, 2020, 6:39am UTC](https://discuss.pytorch.org/t/does-a-dataloader-change-random-state-even-when-shuffle-argument-is-false/92569/2 "2020-08-15T06:39:57Z")

</div>

Based on the description it seems that the test `DataLoader` is calling into the PRNG at one point. Since `shuffle` is set to `False`, the `Dataset` might still use it at one point. Could you post the `Dataset` implementation (for the testset in particular) as well as the test loop?

---

<div class="post-metadata">

**Author:** ![jmaronas](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jmaronas/32/2816_2.png) [@jmaronas](https://discuss.pytorch.org/u/jmaronas)\
**Post date:** [August 17, 2020, 8:20am UTC](https://discuss.pytorch.org/t/does-a-dataloader-change-random-state-even-when-shuffle-argument-is-false/92569/3 "2020-08-17T08:20:31Z")

</div>

This is the dataset class:

```auto

class dataset_class(torch.utils.data.Dataset):
    ''' General dataset class
        Args:
                X: input -> torch.tensor
                Y: targets -> torch.tensor
    '''

    def __init__ (self,X: torch.tensor, Y: torch.tensor):
        super(dataset_class, self). __init__ ()
        assert X is not None , "Invalid None type"
        assert Y is not None, "Invalid None type"
        self.X = X
        self.Y = Y

    def __len__ (self):
        return self.X.size(0)

    def __getitem__ (self,idx):
      
        return self.X[idx],self.Y[idx]

```

where `self.X` and `self.Y` are constant for all the runs, i.e its order is not shuffled unless the dataloader does it. This is the main loop:

```auto
for b,(x,y) in enumerate(train_loader):

     x,y = x.to(cg.device),y.to(cg.device)
                   
      ## loss
      loss = model.loss(x,y)
      optimizer.zero_grad() # just in case
      loss.backward()
      optimizer.step()
              
      # Performance metrics                      
      if ((ep+1)%self.validate_each) == 0:
           model.performance_metrics(x,y,dataset='train')
 
      with torch.no_grad():

          if ((ep+1)%validate_each) == 0:
                    for b,(x,y) in enumerate(test_loader):
                        x,y = x.to(cg.device),y.to(cg.device)
                        model.performance_metrics(x,y,dataset='test')

```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [August 18, 2020, 4:18am UTC](https://discuss.pytorch.org/t/does-a-dataloader-change-random-state-even-when-shuffle-argument-is-false/92569/4 "2020-08-18T04:18:00Z")

</div>

Thanks for the code.  
I can reproduce this behavior using two simple `DataLoaders`:

```python
X_train = torch.arange(10).float().view(-1, 1)
y_train = torch.arange(10).float().view(-1, 1) + 0.1
train_dataset = dataset_class(X_train, y_train)
train_loader = DataLoader(train_dataset, batch_size=2, shuffle=True)

X_val = torch.arange(10, 20).float().view(-1, 1)
y_val = torch.arange(10, 20).float().view(-1, 1) + 0.1
val_dataset = dataset_class(X_val, y_val)
val_loader = DataLoader(val_dataset, batch_size=2, shuffle=False)

seed = 2809
print('Seeding with {}'.format(seed))
torch.manual_seed(seed)
print('Training loop only')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    print('='*10)

seed = 2809
print('Seeding with {}'.format(seed))
torch.manual_seed(seed)
print('Training loop only')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    print('='*10)

seed = 2809
print('Seeding with {}'.format(seed))
torch.manual_seed(seed)
print('Adding validation loop')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    for j, (x_v, y_v) in enumerate(val_loader):
        print('ValIter{}, X_val: {}'.format(j, x_v))
    print('='*10)

```

If you execute the code (with your `dataset_class` definition), you’ll see that the `train_loader` batches are not the same for the second epoch, if the `val_loader` was executed without shuffling.

My best guess is, that the `_BaseDataLoaderIter` calls into the PRNG in [this line of code](https://github.com/pytorch/pytorch/blob/bcab2d68486fbacf9c68674e6b953b407ac09b98/torch/utils/data/dataloader.py#L387), which would be needed to seed each worker [here](https://github.com/pytorch/pytorch/blob/bcab2d68486fbacf9c68674e6b953b407ac09b98/torch/utils/data/dataloader.py#L764-L769).

A workaround would be to create a `torch.Generator` manually and pass it to your `train_loader`, so that PyTorch uses it for the `_base_seed` creation:

```python
gen = torch.Generator()

X_train = torch.arange(10).float().view(-1, 1)
y_train = torch.arange(10).float().view(-1, 1) + 0.1
train_dataset = dataset_class(X_train, y_train)
train_loader = DataLoader(train_dataset, batch_size=2, shuffle=True, generator=gen)

X_val = torch.arange(10, 20).float().view(-1, 1)
y_val = torch.arange(10, 20).float().view(-1, 1) + 0.1
val_dataset = dataset_class(X_val, y_val)
val_loader = DataLoader(val_dataset, batch_size=2, shuffle=False)

seed = 2809
print('Seeding with {}'.format(seed))
#torch.manual_seed(seed)
gen.manual_seed(seed)
print('Training loop only')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    print('='*10)

seed = 2809
print('Seeding with {}'.format(seed))
#torch.manual_seed(seed)
gen.manual_seed(seed)
print('Training loop only')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    print('='*10)

seed = 2809
print('Seeding with {}'.format(seed))
#torch.manual_seed(seed)
gen.manual_seed(seed)
print('Adding validation loop')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    for j, (x_v, y_v) in enumerate(val_loader):
        print('ValIter{}, X_val: {}'.format(j, x_v))
    print('='*10)

```

---

<div class="post-metadata">

**Author:** ![jmaronas](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jmaronas/32/2816_2.png) [@jmaronas](https://discuss.pytorch.org/u/jmaronas)\
**Post date:** [August 19, 2020, 11:46am UTC](https://discuss.pytorch.org/t/does-a-dataloader-change-random-state-even-when-shuffle-argument-is-false/92569/5 "2020-08-19T11:46:30Z")

</div>

Okei thank you. Normally I dont see a difference in performance due to how batches are shuffled. But for my particular problem I am experimenting very different results just by setting validate\_each to different numbers (different numbers implies different number loops over the test dataset hence different order).

Thanks for your solution