# How to effectively load a large text dataset with PyTorch? **URL:** \ **Category:** nlp\ **Created:** [October 15, 2021, 9:23pm UTC](https://discuss.pytorch.org/t/how-to-effectively-load-a-large-text-dataset-with-pytorch/134357 "2021-10-15T21:23:40Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Emanuel\_Huber](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/emanuel_huber/32/31242_2.png) [@Emanuel\_Huber](https://discuss.pytorch.org/u/Emanuel_Huber)\ **Post date:** [October 15, 2021, 9:23pm UTC](https://discuss.pytorch.org/t/how-to-effectively-load-a-large-text-dataset-with-pytorch/134357/1 "2021-10-15T21:23:40Z")
I have hundreds of CSV files that each contain hundreds of megabytes of data. To create a class that inherits from PyTorch’s Dataset the **getitem** method must access a single sample at a time, where the `i` parameter of the function indicates the index of the sample. However, to perform lazy loading my class just saves the name of each file instead of saving all the data in memory. So far so good, but I’m having doubts about how to access a single data in the ` __getitem__ ` method, in my searches I found the following strategies: 1. In ` __init__ ` save the number of samples that each file has, then in **getitem** load the file with pandas and access the respective line. It is possible to know which file to access by the id received as a parameter, for example: having 2 files, where each has 100 samples, id 100 would represent the first sample of the second file, id 101 would represent the second sample of the second file and so on against. However, this approach has an obvious problem which is the number of IOs needed. 2. Perform the same process presented in 1, but store files in a cache as long as it fits in memory. This approach seems to be better than the first as it reduces the number of IOs as some files will be in memory. The problem with this approach is how the caching policy will be carried out, I thought of using something like `dask` to handle this, but `iloc` is quite inefficient according to the [documentation](https://docs.dask.org/en/latest/dataframe-indexing.html). Does anyone have a better idea or indication of some material? ---
**Author:** ![AbdulsalamBande](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abdulsalambande/32/40450_2.png) [@AbdulsalamBande](https://discuss.pytorch.org/u/AbdulsalamBande)\ **Post date:** [October 15, 2021, 9:46pm UTC](https://discuss.pytorch.org/t/how-to-effectively-load-a-large-text-dataset-with-pytorch/134357/2 "2021-10-15T21:46:42Z")
@Emanuel_Huber , why dont you featurizer(embedding or one hot) your text data and store the data in binary format using numpy.memmap() [numpy.memmap — NumPy v1.21 Manual](https://numpy.org/doc/stable/reference/generated/numpy.memmap.html) . Then you can use the normal **getitem** method. ---
**Author:** ![Emanuel\_Huber](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/emanuel_huber/32/31242_2.png) [@Emanuel\_Huber](https://discuss.pytorch.org/u/Emanuel_Huber)\ **Post date:** [October 16, 2021, 5:43pm UTC](https://discuss.pytorch.org/t/how-to-effectively-load-a-large-text-dataset-with-pytorch/134357/3 "2021-10-16T17:43:45Z")
That would be a good solution, but I found a more straightforward one that probably will be integrated on the official PyTorch code, the [webdataset](https://github.com/webdataset/webdataset) package. ---
**Author:** ![Emanuel\_Huber](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/emanuel_huber/32/31242_2.png) [@Emanuel\_Huber](https://discuss.pytorch.org/u/Emanuel_Huber)\ **Post date:** [October 18, 2021, 1:05pm UTC](https://discuss.pytorch.org/t/how-to-effectively-load-a-large-text-dataset-with-pytorch/134357/4 "2021-10-18T13:05:28Z")
After messing with `webdataset` I could not use it for my case. When I convert my 20Gb dataset with webdataset ShardWriter or with `tarp` CLI that conversion generated a 200Gb file and I do not have this space available on my disk. I am not sure, but I think that TFRecord is a better and less disk-consuming approach. After this little journey, my ending was to load all the raw data in memory and tokenized it before passing it to the network. If you have enough RAM, I advise you to do the same.