# How to stream a folder of CSV data for multi-GPU training using DistributedSampler?

**URL:** <https://discuss.pytorch.org/t/how-to-stream-a-folder-of-csv-data-for-multi-gpu-training-using-distributedsampler/182436>\
**Category:** data\
**Created:** [June 19, 2023, 6:30pm UTC](https://discuss.pytorch.org/t/how-to-stream-a-folder-of-csv-data-for-multi-gpu-training-using-distributedsampler/182436 "2023-06-19T18:30:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![shengy90](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shengy90/32/61315_2.png) [@shengy90](https://discuss.pytorch.org/u/shengy90)\
**Post date:** [June 19, 2023, 6:30pm UTC](https://discuss.pytorch.org/t/how-to-stream-a-folder-of-csv-data-for-multi-gpu-training-using-distributedsampler/182436/1 "2023-06-19T18:30:58Z")

</div>

Hi 👋 I have a folder of a lot of CSV files which cannot be loaded into memory. How do I load this data into multi-GPU training?

**What I have done**

1. At the moment, I’m loading this data into my training application using TorchData datapipes ([Example tutorial](https://pytorch.org/data/main/dp_tutorial.html?highlight=parse_csv)).
2. As my full production dataset is too big to fit all into memory and I’d like to accelerate my training, I’d like to train it using multiple GPUs, which each GPU only reading a subset of the data.
3. To do that, I’m trying to use [DistributedSampler](https://pytorch.org/docs/stable/data.html?highlight=distributedsampler#torch.utils.data.distributed.DistributedSampler) alongside with DistributedDataParallel strategy

However, I’m getting the error " MapperIterDataPipe has no _len_" which of course it hasn’t because the full data size is not known in advanced. What’s the best strategy then to stream CSV records into Distributed GPU training in Pytorch?

In tensorflow, there’s a DistributedDataset and DistributedMirrorStrategy which takes care of everything. What’s the best practice for Pytorch?

---

<div class="post-metadata">

**Author:** ![shengy90](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shengy90/32/61315_2.png) [@shengy90](https://discuss.pytorch.org/u/shengy90)\
**Post date:** [June 20, 2023, 8:30am UTC](https://discuss.pytorch.org/t/how-to-stream-a-folder-of-csv-data-for-multi-gpu-training-using-distributedsampler/182436/2 "2023-06-20T08:30:01Z")

</div>

I also attempted to use Pytorch Lightning to do MultiGPU training, and then discovered that Pytorch Lightning doesn’t automatically distribute data across multiple GPU devices for IterableDatasets. Pytorch Lightning only automatically does so for Map Datasets.

Wondering what’s the best practice for production datasets (very large data, sharded across many CSV files) + MultiGPU training in Pytorch!