# How to shuffle an iterable dataset

**URL:** <https://discuss.pytorch.org/t/how-to-shuffle-an-iterable-dataset/64130>\
**Category:** Uncategorized\
**Created:** [December 15, 2019, 6:54pm UTC](https://discuss.pytorch.org/t/how-to-shuffle-an-iterable-dataset/64130 "2019-12-15T18:54:54Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![sharvil](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sharvil/32/21517_2.png) [@sharvil](https://discuss.pytorch.org/u/sharvil)\
**Post date:** [April 30, 2020, 8:30pm UTC](https://discuss.pytorch.org/t/how-to-shuffle-an-iterable-dataset/64130/6 "2020-04-30T20:30:23Z")

</div>

I think the standard approach to shuffling an iterable dataset is to introduce a shuffle buffer into your pipeline. Here’s the class I use to shuffle an iterable dataset:

```python
class ShuffleDataset(torch.utils.data.IterableDataset):
  def __init__ (self, dataset, buffer_size):
    super(). __init__ ()
    self.dataset = dataset
    self.buffer_size = buffer_size

  def __iter__ (self):
    shufbuf = []
    try:
      dataset_iter = iter(self.dataset)
      for i in range(self.buffer_size):
        shufbuf.append(next(dataset_iter))
    except:
      self.buffer_size = len(shufbuf)

    try:
      while True:
        try:
          item = next(dataset_iter)
          evict_idx = random.randint(0, self.buffer_size - 1)
          yield shufbuf[evict_idx]
          shufbuf[evict_idx] = item
        except StopIteration:
          break
      while len(shufbuf) > 0:
        yield shufbuf.pop()
    except GeneratorExit:
      pass

```

You’d wrap your existing iterable dataset like this:

```python
dataset = MyIterableDataset()
dataset = ShuffleDataset(dataset, 1024) # shuffle buffer size depends on your application

```

---

_[View the full topic](https://discuss.pytorch.org/t/how-to-shuffle-an-iterable-dataset/64130)._