# DataLoaders - Multiple files, and multiple rows per column with lazy evaluation

**URL:** <https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769>\
**Category:** Uncategorized\
**Created:** [January 2, 2018, 12:35pm UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769 "2018-01-02T12:35:16Z")\
**Posts on this page:** 1\
**Showing post:** 7

<div class="post-metadata">

**Author:** ![mxlei01](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/aeb1de/32.png) [@mxlei01](https://discuss.pytorch.org/u/mxlei01)\
**Post date:** [January 8, 2018, 12:24am UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/7 "2018-01-08T00:24:00Z")

</div>

I created one dataset for each file, and if there’s only 3000 files then it isn’t that much to hold it inside an array (object that has a reference to).

If you wrote your DataSet with linecache, then it won’t read each file into memory.

At least this is my observation after reading more files than my computer’s memory can support.

---

_[View the full topic](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769)._