# Loading huge data functionality

**URL:** <https://discuss.pytorch.org/t/loading-huge-data-functionality/346>\
**Category:** Uncategorized\
**Created:** [February 5, 2017, 3:46am UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346 "2017-02-05T03:46:02Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Xia\_Yandi](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/73ab20/32.png) [@Xia\_Yandi](https://discuss.pytorch.org/u/Xia_Yandi)\
**Post date:** [February 5, 2017, 3:46am UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/1 "2017-02-05T03:46:03Z")

</div>

Do you have any plan on implementing big data files loading functionality?

Suppose I have 300G data files for training, and I can’t load them all into memory.

For now, I am using TensorFlow, and they provide producer/consumer style data loading:  
[https://www.tensorflow.org/how\_tos/reading\_data/](https://www.tensorflow.org/how_tos/reading_data/)

With it, I don’t need to read all data set into memory at once, and I can load data in parallel fashion.

Any plan on similar functionality?

Thanks.

---

<div class="post-metadata">

**Author:** ![NgPDat](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/7feea3/32.png) [@NgPDat](https://discuss.pytorch.org/u/NgPDat)\
**Post date:** [February 5, 2017, 7:22am UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/2 "2017-02-05T07:22:52Z")

</div>

You can already do that with [Pytorchnet](https://github.com/pytorch/tnt).  
Concretely, you pass a list of data files into `tnt.ListDataset`, then wrap it with `torch.utils.DataLoader`.  
Example code:

```auto
def load_func(line):
    # a line in 'list.txt"

    # Implement how you load a single piece of data here

    # assuming you already load data into src and target respectively
    return {'src': src, 'target': target} # you can return a tuple or whatever you want it to

def batchify(batch):
    # batch will contain a list of {'src', 'target'}, or how you return it in load_func.

    # Implement method to batch the list above into Tensor here

    # assuming you already have two tensor containing batched Tensor for src and target
    return {'src': batch_src, 'target': batch_target} # you can return a tuple or whatever you want it to

dataset = ListDataset('list.txt', load_func) #list.txt contain list of datafiles, one per line
dataset = DataLoader(dataset=dataset, batch_size=50, num_workers=8, collate_fn=batchify) #This will load data when needed, in parallel, up to <num_workers> thread.

for x in dataset: #iterate dataset
    print(x)

```

There are surely other way to do it. Hope this helps.

---

<div class="post-metadata">

**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\
**Post date:** [February 5, 2017, 11:13am UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/3 "2017-02-05T11:13:30Z")

</div>

Just define a `Dataset` object, that only loads a list of files in ` __init__ `, and loads them every time ` __getindex__ ` is called. Then, wrap it in a `torch.utils.DataLoader` with multiple workers, and you’ll have your files loaded lazily in parallel.

```auto
class MyDataset(torch.utils.Dataset):
    def __init__ (self):
        self.data_files = os.listdir('data_dir')
        sort(self.data_files)

    def __getindex__ (self, idx):
        return load_file(self.data_files[idx])

    def __len__ (self):
        return len(self.data_files)

dset = MyDataset()
loader = torch.utils.DataLoader(dset, num_workers=8)

```

---

<div class="post-metadata">

**Author:** ![Xia\_Yandi](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/73ab20/32.png) [@Xia\_Yandi](https://discuss.pytorch.org/u/Xia_Yandi)\
**Post date:** [February 5, 2017, 4:13pm UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/4 "2017-02-05T16:13:04Z")

</div>

That is so cool! Thanks!

---

<div class="post-metadata">

**Author:** ![Morpheus\_Hsieh](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/morpheus_hsieh/32/58960_2.png) [@Morpheus\_Hsieh](https://discuss.pytorch.org/u/Morpheus_Hsieh)\
**Post date:** [February 7, 2017, 10:26am UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/5 "2017-02-07T10:26:48Z")

</div>

Sorry, new to PyTorch. How might one adapt the above method to `pytorch/examples/word_language_model/`? Currently, it seems to load the entire data into cuda, which is causing OOM errors.

---

<div class="post-metadata">

**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\
**Post date:** [February 7, 2017, 1:36pm UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/6 "2017-02-07T13:36:07Z")

</div>

It would require rewriting the whole data loading part. It would need to go over all of it to gather all the tokens, and then lazily load only the batches you request. You’d need to add a proper `torch.utils.Dataset` subclass that does it.

---

<div class="post-metadata">

**Author:** ![Morpheus\_Hsieh](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/morpheus_hsieh/32/58960_2.png) [@Morpheus\_Hsieh](https://discuss.pytorch.org/u/Morpheus_Hsieh)\
**Post date:** [February 8, 2017, 5:58am UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/7 "2017-02-08T05:58:16Z")

</div>

Thanks. I did something that may sound stupid.  
I removed `data.cuda()` from `batchify()` and added  
`data = data.cuda()`  
`target = target.cuda()`  
in `get_batch()`.  
It seems to be running now, but I wonder if this is a quick (and correct) fix.

---

<div class="post-metadata">

**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\
**Post date:** [February 8, 2017, 4:40pm UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/8 "2017-02-08T16:40:37Z")

</div>

Yeah, that’s correct too!

---

<div class="post-metadata">

**Author:** ![Morpheus\_Hsieh](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/morpheus_hsieh/32/58960_2.png) [@Morpheus\_Hsieh](https://discuss.pytorch.org/u/Morpheus_Hsieh)\
**Post date:** [February 9, 2017, 10:48am UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/9 "2017-02-09T10:48:25Z")

</div>

That is a relief! However, I am noticing a huge and growing amount of memory usage now, after running for 40+ hours. Is there anything that I am missing? Perhaps every time when `get_batch()` is run, it is creating `Variable`s and they are kept in memory after the batch is finished? Thanks.

---

<div class="post-metadata">

**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\
**Post date:** [February 9, 2017, 3:07pm UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/10 "2017-02-09T15:07:00Z")

</div>

Is that CPU memory or GPU memory? Everything should get freed from time to time. Are you just running the example with that single modification?

---

<div class="post-metadata">

**Author:** ![Xia\_Yandi](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/73ab20/32.png) [@Xia\_Yandi](https://discuss.pytorch.org/u/Xia_Yandi)\
**Post date:** [February 9, 2017, 9:57pm UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/11 "2017-02-09T21:57:16Z")

</div>

I wrote something follow your instruction. But it doesn’t work for me.

Here is what I do:

```auto
    def _load_hdf5_file(hdf5_file):
        f = h5py.File(hdf5_file, "r")
        data = []
        for key in f.keys():
            data.append(f[key])
        return tuple(data)

    class HDF5Dataset(Dataset):
        def __init__ (self, data_files):
            self.data_files = sorted(data_files)

        def __getitem__ (self, index):
            return _load_hdf5_file(self.data_files[index])

        def __len__ (self):
            return len(self.data_files)

    train_set = HDF5Dataset(train_files) # there is only one file in train_files, i.e. train_files = ["foo_1"]
    train_loader = DataLoader(dataset=train_set,
                              batch_size=train_batch_size,
                              shuffle=True,
                              num_workers=2)

```

And during iteration, I got this error:

```auto
    Traceback (most recent call last):
      File "/usr/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
      File "/usr/lib/python2.7/multiprocessing/util.py", line 207, in __call__
      File "/usr/lib/python2.7/shutil.py", line 239, in rmtree
      File "/usr/lib/python2.7/shutil.py", line 237, in rmtree
    OSError: [Errno 24] Too many open files: '/tmp/pymp-Y6oJsO'
    Process Process-1:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
      File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
      File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 36, in _worker_loop
      File "/usr/lib/python2.7/multiprocessing/queues.py", line 392, in put
      File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/multiprocessing/queue.py", line 17, in send
      File "/usr/lib/python2.7/pickle.py", line 224, in dump
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/pickle.py", line 554, in save_tuple
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/pickle.py", line 606, in save_list
      File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/multiprocessing/forking.py", line 67, in dispatcher
      File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/pickle.py", line 554, in save_tuple
      File "/usr/lib/python2.7/pickle.py", line 286, in save
      File "/usr/lib/python2.7/multiprocessing/forking.py", line 66, in dispatcher
      File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/multiprocessing/reductions.py", line 116, in reduce_storage
      File "/usr/lib/python2.7/multiprocessing/reduction.py", line 145, in reduce_handle
    OSError: [Errno 24] Too many open files

```

And the program never terminates.

Did I do anything wrong? Thanks

By the way, I don’t know if it is appropriate to ask here, how can I post python style code here like you did? Thanks.

---

<div class="post-metadata">

**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\
**Post date:** [February 9, 2017, 11:29pm UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/12 "2017-02-09T23:29:40Z")

</div>

It seems that your system allows only for a small number of open files. Can you try adding `torch.multiprocessing.set_sharing_strategy('file_system')` at the top of your script and try again?

Just append `python` [after the three backticks](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#code) to add syntax highlighting.

---

<div class="post-metadata">

**Author:** ![Xia\_Yandi](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/73ab20/32.png) [@Xia\_Yandi](https://discuss.pytorch.org/u/Xia_Yandi)\
**Post date:** [February 10, 2017, 2:04am UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/13 "2017-02-10T02:04:22Z")

</div>

I added the line, and I got this error:

```auto
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 36, in _worker_loop
    data_queue.put((idx, samples))
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 392, in put
    return send(obj)
  File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/multiprocessing/queue.py", line 17, in send
    ForkingPickler(buf, pickle.HIGHEST_PROTOCOL).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 554, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
    save(x)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
    save(x)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.7/pickle.py", line 639, in _batch_appends
    save(x)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 67, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 554, in save_tuple
    save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 66, in dispatcher
    rv = reduce(obj)
  File "/home/ts-yandixia01/.local/lib/python2.7/site-packages/torch/multiprocessing/reductions.py", line 109, in reduce_storage
    metadata = storage._share_filename_()
RuntimeError: $ Torch: unable to mmap memory: you tried to mmap 0GB. at /home/soumith/local/builder/wheel/pytorch-src/torch/lib/TH/THAllocator.c:317

```

Thanks

---

<div class="post-metadata">

**Author:** ![Morpheus\_Hsieh](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/morpheus_hsieh/32/58960_2.png) [@Morpheus\_Hsieh](https://discuss.pytorch.org/u/Morpheus_Hsieh)\
**Post date:** [February 10, 2017, 5:59am UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/14 "2017-02-10T05:59:54Z")

</div>

That was CPU memory that I noticed using `htop`. Besides the above modification, I used another corpus which is about 9 GB. Anyway, it finished without OOM so I guess it’s fine 😛.

---

<div class="post-metadata">

**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\
**Post date:** [February 10, 2017, 2:22pm UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/15 "2017-02-10T14:22:10Z")

</div>

That was probably an out of memory error. If the data or the code is public (or if you could just isolate the data loading into a separate script), I could run it myself and make sure it doesn’t leak. But we have some tests for that.

---

<div class="post-metadata">

**Author:** ![Xia\_Yandi](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/73ab20/32.png) [@Xia\_Yandi](https://discuss.pytorch.org/u/Xia_Yandi)\
**Post date:** [February 11, 2017, 1:33am UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/16 "2017-02-11T01:33:44Z")

</div>

Hi,

I just wrote a simple demo code to reproduce the error. The code is here:

> **[xiayandi/Pytorch\_text\_classification](https://github.com/xiayandi/Pytorch_text_classification)**
>
> A simple implementation of CNN based text classification in Pytorch - xiayandi/Pytorch\_text\_classification

The data is randomly generated, but everything is the same as mine except the actual values.

You could run main.py to reproduce the error.

I don’t know if this will be useful, my system is:  
Ubuntu 16.04, TitanX, cuda 8.0, pip installed pytorch

Thanks!

---

<div class="post-metadata">

**Author:** ![Morpheus\_Hsieh](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/morpheus_hsieh/32/58960_2.png) [@Morpheus\_Hsieh](https://discuss.pytorch.org/u/Morpheus_Hsieh)\
**Post date:** [February 11, 2017, 4:27pm UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/17 "2017-02-11T16:27:19Z")

</div>

Thanks so much. I literally just modified a very small part of the original example code `main.py`

```auto
--- a/word_language_model/main.py
+++ b/word_language_model/main.py
@@ -57,8 +57,9 @@ def batchify(data, bsz):
     nbatch = data.size(0) // bsz
     data = data.narrow(0, 0, nbatch * bsz)
     data = data.view(bsz, -1).t().contiguous()
- if args.cuda:
- data = data.cuda()
+
+ # if args.cuda:
+ # data = data.cuda()
     return data
 
 eval_batch_size = 10
@@ -103,6 +104,9 @@ def get_batch(source, i, evaluation=False):
     seq_len = min(args.bptt, len(source) - 1 - i)
     data = Variable(source[i:i+seq_len], volatile=evaluation)
     target = Variable(source[i+1:i+1+seq_len].view(-1))
+ if args.cuda:
+ data = data.cuda()
+ target = target.cuda()
     return data, target

```

---

<div class="post-metadata">

**Author:** ![apaszke](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/32/21_2.png) [@apaszke](https://discuss.pytorch.org/u/apaszke)\
**Post date:** [February 11, 2017, 4:58pm UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/18 "2017-02-11T16:58:19Z")

</div>

I’ve started a script running @Morpheus_Hsieh’s script, I’ll try the demo later. Thanks!

---

<div class="post-metadata">

**Author:** ![itzjustricky](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/45deac/32.png) [@itzjustricky](https://discuss.pytorch.org/u/itzjustricky)\
**Post date:** [May 12, 2017, 11:20pm UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/19 "2017-05-12T23:20:14Z")

</div>

for the code snippet that you provided.

```
class MyDataset(torch.utils.Dataset):
    def __init__ (self):
        self.data_files = os.listdir('data_dir')
        sort(self.data_files)

    def __getindex__ (self, idx):
        return load_file(self.data_files[idx])

    def __len__ (self):
        return len(self.data_files)

```

Are the file paths stored in self.data\_files suppose to represent each batch of data (or data per loop) returned by iterating loader?

---

<div class="post-metadata">

**Author:** ![smth](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/32/13_2.png) [@smth](https://discuss.pytorch.org/u/smth)\
**Post date:** [May 13, 2017, 5:42pm UTC](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/20 "2017-05-13T17:42:03Z")

</div>

it is data per instance of the loop

[Next page](https://discuss.pytorch.org/t/loading-huge-data-functionality/346.md?page=2)
