# DataLoaders - Multiple files, and multiple rows per column with lazy evaluation

**URL:** <https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769>\
**Category:** Uncategorized\
**Created:** [January 2, 2018, 12:35pm UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769 "2018-01-02T12:35:16Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![mxlei01](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/aeb1de/32.png) [@mxlei01](https://discuss.pytorch.org/u/mxlei01)\
**Post date:** [January 2, 2018, 12:35pm UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/1 "2018-01-02T12:35:16Z")

</div>

Hi All,

I’m trying to create a DataSet class that can load many large file, and each file have rows of data a model would need to train.

I’ve read: [Loading huge data functionality](https://discuss.pytorch.org/t/loading-huge-data-functionality/346/3)

```
class MyDataset(torch.utils.Dataset):
    def __init__ (self):
        self.data_files = os.listdir('data_dir')
        sort(self.data_files)

    def __getindex__ (self, idx):
        return load_file(self.data_files[idx])

    def __len__ (self):
        return len(self.data_files)

dset = MyDataset()
loader = torch.utils.DataLoader(dset, num_workers=8)

```

Essentially if each file is an image, then you can use this functionality to only load the required images in memory.

So what about a file with multiple lines? Then I came across this github thread: [https://github.com/pytorch/text/issues/130](https://github.com/pytorch/text/issues/130)

```
class LazyTextDataset(Dataset):
    def __init__ (self, filename):
        self._filename = filename
        self._total_data = 0
        with open(filename, "r") as f:
            self._total_data = len(f.readlines()) - 1

    def __getitem__ (self, idx):
        line = linecache.getline(self._filename, idx + 1)
        csv_line = csv.reader([line])
        .....

    def __len__ (self):
        return self._total_data

```

Now the problem I have is how do I combine these two so that I can load multiple files that are all large?

Since ` __getitem__ ` only uses 1 idx, there isn’t a way to load multiple file and each of their lines?

Can someone point me to the right direction?

---

<div class="post-metadata">

**Author:** ![chenyuntc](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenyuntc/32/299_2.png) [@chenyuntc](https://discuss.pytorch.org/u/chenyuntc)\
**Post date:** [January 2, 2018, 1:09pm UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/2 "2018-01-02T13:09:24Z")

</div>

`torch.utils.data.ConcatDataset` maybe helpful

[http://pytorch.org/docs/0.3.0/data.html#torch.utils.data.ConcatDataset](http://pytorch.org/docs/0.3.0/data.html#torch.utils.data.ConcatDataset)

---

<div class="post-metadata">

**Author:** ![mxlei01](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/aeb1de/32.png) [@mxlei01](https://discuss.pytorch.org/u/mxlei01)\
**Post date:** [January 2, 2018, 2:07pm UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/3 "2018-01-02T14:07:52Z")

</div>

I was looking at ConcatDataset too, but one of my question is does it support shuffle between datasets?

Let’s say I have 2 datasets, A, and B.

Can data be shuffled from A to B, and B to A? From the code (I don’t understand the code 100%), it seems that data within their own dataset are shuffled.

---

<div class="post-metadata">

**Author:** ![chenyuntc](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenyuntc/32/299_2.png) [@chenyuntc](https://discuss.pytorch.org/u/chenyuntc)\
**Post date:** [January 2, 2018, 3:20pm UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/4 "2018-01-02T15:20:05Z")

</div>

> [@mxlei01](#):
>
> Can data be shuffled from A to B, and B to A

yes

shuffle was performed by dataloader.

---

<div class="post-metadata">

**Author:** ![mxlei01](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/aeb1de/32.png) [@mxlei01](https://discuss.pytorch.org/u/mxlei01)\
**Post date:** [January 2, 2018, 5:39pm UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/5 "2018-01-02T17:39:42Z")

</div>

Gotcha, I’ll try to use ConcatDataset.

Thanks for your help. Really appreciated your help!

---

<div class="post-metadata">

**Author:** ![adityagourav](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adityagourav/32/1939_2.png) [@adityagourav](https://discuss.pytorch.org/u/adityagourav)\
**Post date:** [January 7, 2018, 9:28pm UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/6 "2018-01-07T21:28:18Z")

</div>

So, does it mean I need to create a Dataset for every file in the directory? I have like 118 million train rows split in 3000 files.

Assuming there are N rows in each file (which now is Dataset), will the ConcatDatastet read individual Datasets into memory?

---

<div class="post-metadata">

**Author:** ![mxlei01](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/aeb1de/32.png) [@mxlei01](https://discuss.pytorch.org/u/mxlei01)\
**Post date:** [January 8, 2018, 12:24am UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/7 "2018-01-08T00:24:00Z")

</div>

I created one dataset for each file, and if there’s only 3000 files then it isn’t that much to hold it inside an array (object that has a reference to).

If you wrote your DataSet with linecache, then it won’t read each file into memory.

At least this is my observation after reading more files than my computer’s memory can support.

---

<div class="post-metadata">

**Author:** ![praateekmahajan](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/praateekmahajan/32/2228_2.png) [@praateekmahajan](https://discuss.pytorch.org/u/praateekmahajan)\
**Post date:** [July 1, 2019, 9:24pm UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/8 "2019-07-01T21:24:00Z")

</div>

```auto
class LazyTextDataset(Dataset):
    def __init__ (self, filename):
        self._filename = filename
        self._total_data = int(subprocess.check_output("wc -l " + filename, shell=True).split()[0]) - 1

    def __getitem__ (self, idx):
        line = linecache.getline(self._filename, idx + 1)
        csv_line = csv.reader([line])
        return next(csv_line)
      
    def __len__ (self):
        return self._total_data

path = /where_csv_files_are_dumped/
files = list(map(lambda x : path + x, (filter(lambda x : x.endswith("csv"), os.listdir(path)))))
datasets = list(map(lambda x : LazyTextDataset(x), files))
dataset = ConcatDataset(datasets)

```

This should work for multiple csv files where each row is represented by a training example.

---

<div class="post-metadata">

**Author:** ![Santosh-Gupta](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh-gupta/32/30716_2.png) [@Santosh-Gupta](https://discuss.pytorch.org/u/Santosh-Gupta)\
**Post date:** [November 3, 2019, 3:30am UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/9 "2019-11-03T03:30:22Z")

</div>

I am going to be working with many files, that all can’t fit into memory at the same time.

Since the data is text that I would like to be loaded quickly, I was hoping to use ether pytorch.load/pickle files, or pandas/parquet files.

However, it looks like `ConcatDatastet` can’t work for my case, since there’s no way to load specific lines only, like with csv files and linecache.getline, so I’ll have to load all the files at once.

So now I’m thinking I’ll just load a fraction of the files, as much as will fit in memory, and do a `ConcatDatastet` for those files. Perhaps using pytorch multiprocessing to load those several files at once. And then use `ConcatDatastet` of that chunk.

I’ll probably have to code additional shuffling code for shuffling among the chunks.

Does this strategy sound like the best approach? Or are there some additional Pytorch tools that can help me around this?

---

<div class="post-metadata">

**Author:** ![praateekmahajan](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/praateekmahajan/32/2228_2.png) [@praateekmahajan](https://discuss.pytorch.org/u/praateekmahajan)\
**Post date:** [February 18, 2020, 7:40pm UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/10 "2020-02-18T19:40:54Z")

</div>

Hey @Santosh-Gupta, if you’re looking for lazy loading for parquet files I’d recommend exploring [https://github.com/vahidk/tfrecord](https://github.com/vahidk/tfrecord) or [https://github.com/uber/petastorm](https://github.com/uber/petastorm).

- TFRecord would require you to go from parquet to tfrecord.
- Petastorm should work out of the box.

---

<div class="post-metadata">

**Author:** ![p568912](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/edb3f5/32.png) [@p568912](https://discuss.pytorch.org/u/p568912)\
**Post date:** [March 25, 2020, 3:38am UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/11 "2020-03-25T03:38:48Z")

</div>

Sorry !I’m new to pytorch.I want to check my guess is right.  
while calling **getitem** in LazyTextDataset ,it will generate I/O one time.  
For example, if my training dataset is 40000 ,it will generate I/O 40000 time.  
Is it time-consuming job while we have to read training data?

---

<div class="post-metadata">

**Author:** ![matthewleigh](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matthewleigh/32/60035_2.png) [@matthewleigh](https://discuss.pytorch.org/u/matthewleigh)\
**Post date:** [March 9, 2021, 9:32am UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/13 "2021-03-09T09:32:23Z")

</div>

I have been trying to use this method to go through my dataset of around one thousand csv files, each containing about 300k lines (one training sample per line).

This method leads to a slow buildup of CPU memory as I iterate through the batches until it crashes.

I have tried putting in linecache.clearcache() at the end of each batch but it isnt clearing up the memory.

---

<div class="post-metadata">

**Author:** ![matthewleigh](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matthewleigh/32/60035_2.png) [@matthewleigh](https://discuss.pytorch.org/u/matthewleigh)\
**Post date:** [March 9, 2021, 9:35am UTC](https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/14 "2021-03-09T09:35:26Z")

</div>

Did you find a solution to this? It seems like exactly what I am trying to implement.