# Dataloader for partitioned dataset **URL:** \ **Category:** data\ **Created:** [April 21, 2022, 10:15pm UTC](https://discuss.pytorch.org/t/dataloader-for-partitioned-dataset/149777 "2022-04-21T22:15:39Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![iknunyants](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iknunyants/32/48842_2.png) [@iknunyants](https://discuss.pytorch.org/u/iknunyants)\ **Post date:** [April 21, 2022, 10:15pm UTC](https://discuss.pytorch.org/t/dataloader-for-partitioned-dataset/149777/1 "2022-04-21T22:15:39Z")
I have a large dataframe dataset. There are float columns but also columns with lists of int. I’ve partitioned it into smaller files. Tried custom dataset which gets [filename, index] in getitem and loads this file if not already loaded. But when I set num\_workers \> 1, ram used by processes keeps increasing sufficiently. What is the efficient way to construct dataloader which can read files one by one? ---
**Author:** ![ejguan](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/5f8ce5/32.png) [@ejguan](https://discuss.pytorch.org/u/ejguan)\ **Post date:** [April 25, 2022, 8:08pm UTC](https://discuss.pytorch.org/t/dataloader-for-partitioned-dataset/149777/2 "2022-04-25T20:08:30Z")
Could you share code snippet? ---
**Author:** ![iknunyants](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iknunyants/32/48842_2.png) [@iknunyants](https://discuss.pytorch.org/u/iknunyants)\ **Post date:** [April 25, 2022, 8:47pm UTC](https://discuss.pytorch.org/t/dataloader-for-partitioned-dataset/149777/3 "2022-04-25T20:47:27Z")
Here is my code in custom sampler. In self.indices[i][j], i - parquet file, j - set of indicies ```auto def __iter__ (self): all_batches = [] for j, file in enumerate(self.datafiles): batches = [] for inds in self.indices[j]: sampler = BatchSampler(SubsetRandomSampler(inds), batch_size=self.batch_size, drop_last=self.drop_last) for sample in sampler: batches.append(sample) shuffle(batches) all_batches.extend(batches) for batch in all_batches: yield batch ``` Here is what I do in my custom dataset getitem ```auto def __getitem__ (self, idx: Tuple[str, int]) -> Tuple: file, idx = idx if self.cur_file != file: self.cur_file = file del self.data self.data = pd.read_parquet(self.cur_file) gc.collect() credit = self.data.iloc[idx].copy() ``` ---
**Author:** ![ejguan](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/5f8ce5/32.png) [@ejguan](https://discuss.pytorch.org/u/ejguan)\ **Post date:** [April 26, 2022, 2:34pm UTC](https://discuss.pytorch.org/t/dataloader-for-partitioned-dataset/149777/4 "2022-04-26T14:34:31Z")
I am curious about the size of each parquet file. When you run multiple workers, the RAM usage would be `num_of_workers` \* `parquet_size` in total. In order to reduce the amount of memory, either reduce `num_of_worker` or reduce `parquet_size` would be the best way. ---
**Author:** ![iknunyants](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iknunyants/32/48842_2.png) [@iknunyants](https://discuss.pytorch.org/u/iknunyants)\ **Post date:** [April 26, 2022, 2:49pm UTC](https://discuss.pytorch.org/t/dataloader-for-partitioned-dataset/149777/5 "2022-04-26T14:49:57Z")
I agree, this is one way. But are there any solutions. Alternatives to pandas dataframe? Other structure of my custom dataset? I’m open to your ideas 🙂 ---
**Author:** ![ejguan](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/5f8ce5/32.png) [@ejguan](https://discuss.pytorch.org/u/ejguan)\ **Post date:** [April 28, 2022, 7:59pm UTC](https://discuss.pytorch.org/t/dataloader-for-partitioned-dataset/149777/6 "2022-04-28T19:59:48Z")
IIUC, We currently do have a solution to stream parquet file using `TorchData`. But, it requires you to run pipeline in Iterative-style using `DataPipe`. See: [data/dataframemaker.py at b6ade8f097bc9ac08460cd403034a35daff09cfa · pytorch/data · GitHub](https://github.com/pytorch/data/blob/b6ade8f097bc9ac08460cd403034a35daff09cfa/torchdata/datapipes/iter/util/dataframemaker.py#L82-L83) cc: @nivek ---
**Author:** ![ejguan](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/5f8ce5/32.png) [@ejguan](https://discuss.pytorch.org/u/ejguan)\ **Post date:** [April 28, 2022, 8:40pm UTC](https://discuss.pytorch.org/t/dataloader-for-partitioned-dataset/149777/7 "2022-04-28T20:40:10Z")
> [@iknunyants](#): > > ```auto > credit = self.data.iloc[idx].copy() > > ``` Besides, I am curious about `credit` object. Do you de-reference it later in your training loop? ---
**Author:** ![iknunyants](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iknunyants/32/48842_2.png) [@iknunyants](https://discuss.pytorch.org/u/iknunyants)\ **Post date:** [May 2, 2022, 2:15pm UTC](https://discuss.pytorch.org/t/dataloader-for-partitioned-dataset/149777/8 "2022-05-02T14:15:56Z")
I do, yes. I’m taking column values from it. Thanks for mentioning TorchData, I’ll try it later ---
**Author:** ![Dushan](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/ee59a6/32.png) [@Dushan](https://discuss.pytorch.org/u/Dushan)\ **Post date:** [July 11, 2025, 3:26am UTC](https://discuss.pytorch.org/t/dataloader-for-partitioned-dataset/149777/9 "2025-07-11T03:26:57Z")
This seems to be the best way to handle large partitioned datasets that I can find after a lot of searching. Im surprised there isn’t a native solution in pytorch, or more examples out there, since I thought NLP datasets would be too big for memory by default, and text doesn’t fit into the vision framework of seperate files for each training sample. Is there something I’m missing? Do I really have to write code my self to track (file, index). Surely there’s a library that abstracts this detail away for me.