# DataLoader memory usage keeps increasing

**URL:** <https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461>\
**Category:** data\
**Created:** [April 8, 2024, 5:09pm UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461 "2024-04-08T17:09:59Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![tobiasr](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/58f4c7/32.png) [@tobiasr](https://discuss.pytorch.org/u/tobiasr)\
**Post date:** [April 8, 2024, 5:09pm UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461/1 "2024-04-08T17:09:59Z")

</div>

Hello,  
i am trying to use pytorchs Dataset and DataLoader to load a large dataset of several 100GB. This is of course too large to be stored in RAM, so parallel, lazy loading is needed. I am trying to load one large HDF file with a combination of a custom Dataset and the DataLoader. This happens on a cluster where the submission of jobs is done with HT Condor. Here i can request an amount of RAM for my Jobs.  
However when i try to load data it will always run out of allocated memory and the job will be terminated regardless of how much memory i request. When checking the log of my job i can see that the used RAM is continuously increasing until it overshoots the allocated memory at some point and the job is killed. Of course i have tried to increase the requested memory, however the ram will still overflow at some point.

The Dataset code:

```
class DataSet(torch.utils.data.Dataset):
    def __init__ (self, file_name, file_dir):
        path = os.path.join(file_dir, file_name)
        file = h5py.File(path, 'r')
        self.labels = file['labels']
        self.specs = file['spectrogram']
        # some irrelevant lines defining self.means, self.std, ...

    def __len__ (self):
        return len(self.labels)

    def __getitem__ (self, idx):
        label = self.labels[idx]
        label = torch.from_numpy(label)
        label = (label - self.means) / self.stds            

        spec = self.specs[idx, ...]
        spec = np.array(spec).astype(float)
        spec = (torch.from_numpy(spec) - self.specmean) / self.specstd
        spec = torch.unsqueeze(spec, dim = 0) # feature dimension (convolutional layer)
    
        return label, spec

```

the main code i have is:

```
train_set = DataSet(
    filename, 
    data_directory
)

train_loader = torch.utils.data.DataLoader(
    train_set,
    batch_size=256, 
    shuffle=True, 
    drop_last=True, 
    num_workers=12, 
    pin_memory=False
)

for index, (labels, conditions) in enumerate(train_loader):
    labels, conditions = labels.to('cuda'), conditions.to('cuda')
    del labels, conditions

```

The for loop here is empty since i am only trying to monitor the loading of my data.

So i have tried to generate some smaller files and tried to see what happens when i try to load these. I have one file of ~7GB and one file of ~70GB, which i loaded with the same code. The iteration over the 7GB takes around 3s and it does not run out of memory. It also uses more RAM than i would expect. On one run it had a memory usage of almost 8GB. This is more than the actual file, implying to me that the whole file is loaded into RAM. This is what i particularly wanted to avoid and it would explain why the job on the larger files fails.

The 70Gb file will not be loaded again, even when i requested more than 70GB RAM (probably it would be loaded at some point, but i did not want to waste even more time and resources watching the used memory increase).  
When i instead wrap another range(10) for loop around the loading of the 7GB file, this will work. It also still runs fairly quickly with ~30s. But the used RAM went up to 36GB in this particular run. This is of course way larger than the file itself, which should not happen! This is some other memory leak. I can fix this by running gc.collect() after each time i load the file in the for loop, but i would prefer if the leak were not there at all.

Some remarks:

- When opening the HDF files datasets with file[‘dataset’] the data will not be loaded yet. Only when i call an event with file[‘dataset’][index] the actual data will be loaded. (I have also checked this using profile from memory\_profiler on **init** () of the dataset)
- I am using chunking in my HDF files
- pin\_memory = False seems to be the better choice for me (faster, less memory usage)
- reducing the batch size did not seem to help
- removing the normalizing steps from **getitem** () does not stop the memory leak

This seems to be an issue with large files and some kind of memory leak in my code, which i could not find yet. To me it seems like the code tries to load the entire file at some point. I have also seen some other posts here which show similar problems, however i did not see any solution yet. If you can spot my mistake i would be very happy. An alternative solution might be to just write many small files and load them together with a DataLoader, but i could not make this work either so far. So if this works i would also appreciate some pointers. I have been stuck on this way to long now, so any help would be much appreciated!

---

<div class="post-metadata">

**Author:** ![divinho](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/9dc877/32.png) [@divinho](https://discuss.pytorch.org/u/divinho)\
**Post date:** [April 8, 2024, 6:09pm UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461/2 "2024-04-08T18:09:44Z")

</div>

One issue I see in your code is that each worker is going to use the entire dataset (it should use a subset). But that doesn’t explain the increasing memory.

There may be something going on with hdf5. The classic reason for this to happen is because of using lists to store data, see this issue: [DataLoader num\_workers \> 0 causes CPU memory from parent process to be replicated in all worker processes · Issue #13246 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/issues/13246)  
Doesn’t seem like you’re doing that though.

this is a good blogpost [Demystify RAM Usage in Multi-Process Data Loaders - Yuxin's Blog](https://ppwwyyxx.com/blog/2022/Demystify-RAM-Usage-in-Multiprocess-DataLoader/)

---

<div class="post-metadata">

**Author:** ![tobiasr](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/58f4c7/32.png) [@tobiasr](https://discuss.pytorch.org/u/tobiasr)\
**Post date:** [April 10, 2024, 4:00pm UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461/3 "2024-04-10T16:00:48Z")

</div>

Thanks for the answer. However the blog post and this github discussion both start with big lists/arrays as a dataset, which are already loaded into memory completely, if im not mistaken. That is something which technically should not happen with a hdf file. I would think the copying from the multiprocessing mentioned in this blog post is still probably happening and filling my RAM though. So i might be able to fix this. My first attempt was to go to num\_workers=0 to avoid multiprocessing, but that did not seem to help much. I still ran into memory consumption on the order of the file size.  
What do you mean with the subdatasets? Should i make subdataset objects from torch.utils.data…? Should i create one DataLoader with one worker for each of these subsets?

---

<div class="post-metadata">

**Author:** ![divinho](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/9dc877/32.png) [@divinho](https://discuss.pytorch.org/u/divinho)\
**Post date:** [April 12, 2024, 2:59pm UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461/4 "2024-04-12T14:59:11Z")

</div>

Ignore my comment about subsetting data, I was wrong.

If you still get the memory consumption with num\_workers=0 then it’s not related to the issues I mentioned. I would check your other code for whether it’s using more data as training continues.

---

<div class="post-metadata">

**Author:** ![rrrongon](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rrrongon/32/69629_2.png) [@rrrongon](https://discuss.pytorch.org/u/rrrongon)\
**Post date:** [April 28, 2024, 9:53pm UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461/5 "2024-04-28T21:53:57Z")

</div>

I am experiencing the same issue. I created my custom dataset where I read a 2GB data that I want to keep in memory for the entire program having 5 epochs. I tried two version of approach.

Version 1: custom dataset gets initialized with the storage path. I load the data from storage. This program takes less than 1GB of host memory.  
Version 2: Custom dataset loads the entire data into memory while initializing the dataset. Then I pass the dataset to Datalaoder. I allocated 6GB of memory.  
In Version 2, during 2nd epoch I ran out of memory. I checked the initialization happens only once though.

I also ran out of memory even I am loading the data in my customDataset - it reaches to memory limit. Code structure is as bellow:

```auto
def run(rank, size, loss_fn, comm, imagenet_config):

    #Initializing dataset and dataloader 
    dataset = CustomDataset()
           # Here I am reding the file. Not even accessing the data in __get_item()__
   epoch_no = 3
   dataloader = DataLoader()
   for epoch in range(epoch_no):
        for data, label in Dataloader:
            continue

   # Using the DataLoader and Dataset. __get_item__ (), I am not even accessing data that I loaded in Dataset. That means, Loading the file is only when Dataset class is initialized and I ensured that using printing statements.

```

In this setup I run out of memory. To get rid of confusions, I used one worker, I tried to read file in different way and location of file.

- I tried to read data out of customDataset, in another class and was accessing loaded data from **get\_item()** using an instance of the dataLoaded Class. Did not work. Memory Limit issue.

Please let me know if anyone has solved this situation before.

Thanks

---

<div class="post-metadata">

**Author:** ![rrrongon](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rrrongon/32/69629_2.png) [@rrrongon](https://discuss.pytorch.org/u/rrrongon)\
**Post date:** [April 29, 2024, 4:46pm UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461/6 "2024-04-29T16:46:55Z")

</div>

[Dataloader’s memory usage keeps increasing during one single epoch. · Issue #20433 · pytorch/pytorch (github.com)](https://github.com/pytorch/pytorch/issues/20433)

This discussion is the one probably you that can help you fixing the issue. It says, Except

- Replace list with a numpy array
- Wrap the list in `multiprocessing.Manager`
- Encode a list of strings in a numpy array of integers

This three data structures, every thing else create a copy on access, that drastically increases the memory. This, is a Pytorch side memory issue. I hope they fix it soon.

So, converting your hdf5 data to any one of these structures can solve the problem I believe. I solved mine, converting PIL.image to numpy array. In my case, I had to convert each image to np array and enclosed each np.array to another final np.array. If I keep anyone of these as list, memory issue occurs.

I hope, this will solve your problem too.

Thanks.

---

<div class="post-metadata">

**Author:** ![tobiasr](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/58f4c7/32.png) [@tobiasr](https://discuss.pytorch.org/u/tobiasr)\
**Post date:** [May 3, 2024, 9:41am UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461/7 "2024-05-03T09:41:59Z")

</div>

An update on this:  
I am currently dealing with several issues. The DataLoader “memory leak” can apparently be fixed like one of the comments suggests. Especially the blogpost Demystify RAM Usage … highlighted in one of the comments is a helpful contribution. However i could not confirm this, since i have an additional issue with the storage of my data, which i will not go into here. My current solution for this is just a denser dataset which fits into my RAM directly, without any usage of the DataLoader. This works very well, however at some point i may come back here with some additional comments on larger datasets.

---

<div class="post-metadata">

**Author:** ![Jose-Verdu-Diaz](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jose-verdu-diaz/32/71050_2.png) [@Jose-Verdu-Diaz](https://discuss.pytorch.org/u/Jose-Verdu-Diaz)\
**Post date:** [June 27, 2024, 3:10pm UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461/8 "2024-06-27T15:10:05Z")

</div>

I’m running with the same problem. My Dataset class has multiple lists with file paths instead of the actual data to avoid using too much memory. The **getitem** method loads the indexed file, and applies a set of transformations (crop, resize, augment, …).

I encapsulated all the python lists in numpy arrays, but I still encounter a “memory leak”. Should I also encapsulate python dictionaries, floats, bools, … in summary, any native python type?

---

<div class="post-metadata">

**Author:** ![Yuyang\_Hu](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuyang_hu/32/61368_2.png) [@Yuyang\_Hu](https://discuss.pytorch.org/u/Yuyang_Hu)\
**Post date:** [July 3, 2024, 2:37pm UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461/9 "2024-07-03T14:37:26Z")

</div>

I got the same problem. Did you fix it? Thanks.

---

<div class="post-metadata">

**Author:** ![Jose-Verdu-Diaz](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jose-verdu-diaz/32/71050_2.png) [@Jose-Verdu-Diaz](https://discuss.pytorch.org/u/Jose-Verdu-Diaz)\
**Post date:** [August 6, 2024, 1:39pm UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461/10 "2024-08-06T13:39:11Z")

</div>

I managed to fix it. Here you have a beautiful summary of strategies to solve this issue:

> <https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662>
>
> \# Editor note: There is a known workaround further down on this issue, which is …to NOT use Python lists, but instead using something else, e.g., torch.tensor directly. See https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 . You can use a numpy array, but it only fixes the issue for the \`fork\` start method. See https://github.com/pytorch/pytorch/issues/13246#issuecomment-1364587359 for more details
> 
> 
> \## 🐛 Bug
> 
> CPU memory will leak if the DataLoader \`num\_workers \> 0\`.
> 
> \## To Reproduce
> Run the following snippet:
> 
> \`\`\`
> from torch.utils.data import Dataset, DataLoader
> from PIL import Image
> from torchvision import transforms
> import os
> 
> class DataIter(Dataset):
> def \_\_init\_\_(self):
> path = "path/to/data"
> self.data = \[\]
> 
> for cls in os.listdir(path):
> for img in os.listdir(os.path.join(path, cls)):
> self.data.append(os.path.join(path, cls, img))
> 
> def \_\_len\_\_(self):
> return len(self.data)
> 
> def \_\_getitem\_\_(self, idx):
> with Image.open(self.data\[idx\]) as img:
> img = img.convert('RGB')
> return transforms.functional.to\_tensor(img)
> 
> 
> train\_data = DataIter()
> train\_loader = DataLoader(train\_data, batch\_size=300,
> shuffle=True,
> drop\_last=True,
> pin\_memory=False,
> num\_workers=18)
> 
> for i, item in enumerate(train\_loader):
> if i % 200 == 0:
> print(i)
> \`\`\`
> \## Expected behavior
> 
> CPU memory will gradually start increasing, eventually filling up the whole RAM. E.g., the process starts with around 15GB and fills up the whole 128GB available on the system. 
> When the \`num\_workers=0\`, RAM usage is constant.
> 
> \## Environment
> 
> \`\`\`
> PyTorch version: 1.0.0.dev20181028
> Is debug build: No
> CUDA used to build PyTorch: 9.0.176
> 
> OS: Ubuntu 16.04.4 LTS
> GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
> CMake version: version 3.5.1
> 
> Python version: 3.5
> Is CUDA available: Yes
> CUDA runtime version: 9.0.176
> GPU models and configuration: 
> GPU 0: GeForce GTX 1080 Ti
> GPU 1: GeForce GTX 1080 Ti
> GPU 2: GeForce GTX 1080 Ti
> 
> Nvidia driver version: 390.67
> cuDNN version: Probably one of the following:
> /usr/lib/x86\_64-linux-gnu/libcudnn.so.7.1.4
> 
> Versions of relevant libraries:
> \[pip\] Could not collect
> \[conda\] Could not collect
> 
> PIL.\_\_version\_\_
> '5.3.0'
> \`\`\`
> 
> \## Additional info
> There are around 24 million images in the dataset and all image paths are loaded into a single list as presented in the above code snippet.
> 
> I have also tried multiple Pytorch (0.4.0 and 0.4.1) versions and the effect is the same.
> 
> cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @SsnL @VitalyFedyunin @ejguan

TLDR: This issue is not a pytorch memory leak, but an issue with how Python shares variables when using multiprocessing causing the Dataset variables to be copied in memory over and over, eventually leading to an OOM error.

I solved it by converting all lists in the Dataset to numpy arrays. Lists with strings required additional processing.

Additionally, I converted all native Python dictionaries into a pandas DataFrame as follows. Please note that, in my case, dictionaries only contain integers (both for keys and values) and I don’t know if further wrangling is necessary for dictionaries with strings:

```py
class CustomDataset(Dataset):
   def __init__ (self, paths:list, ...):
      self.paths = np.array(paths).astype(np.string_)
      self.example_dict = pd.DataFrame({k: pd.Series(v) for k, v in example_dict.items()}, dtype=np.float16)
      ...

   def __getitem__ (self, idx:int):
      paths = [str(s, encoding="utf-8") for s in self.paths]

```

---

<div class="post-metadata">

**Author:** ![ikarus5991](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ikarus5991/32/72452_2.png) [@ikarus5991](https://discuss.pytorch.org/u/ikarus5991)\
**Post date:** [September 7, 2024, 5:37am UTC](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461/11 "2024-09-07T05:37:21Z")

</div>

I faced the similar problem recently. So I think the problem is you open h5 file just once in **init**. So i would recommend you create custom worker\_init\_fn where you open file in each worker separately.

```python
class DataSet(torch.utils.data.Dataset):
    def __init__ (self, file_name, file_dir):
        self.path = os.path.join(file_dir, file_name)

    def __len__ (self):
        return len(self.labels)

    def __getitem__ (self, idx):
        label = self.labels[idx]
        label = torch.from_numpy(label)
        label = (label - self.means) / self.stds            

        spec = self.specs[idx, ...]
        spec = np.array(spec).astype(float)
        spec = (torch.from_numpy(spec) - self.specmean) / self.specstd
        spec = torch.unsqueeze(spec, dim = 0) # feature dimension (convolutional layer)
    
        return label, spec
def worker_init_fn(worker_id):
    worker_info = torch.utils.data.get_worker_info()
    dataset = worker_info.dataset
    h5file = h5py.File(dataset.items_path, 'r'):
    dataset.labels = h5file['labels']
    dataset.specs = h5file['spectrogram']

train_set = DataSet(
    filename, 
    data_directory
)

train_loader = torch.utils.data.DataLoader(
    train_set,
    batch_size=256, 
    shuffle=True, 
    drop_last=True, 
    num_workers=12, 
    pin_memory=False,
   worker_init_fn=worker_init_fn
)

```