Hello,
When using IterableDataset: For instance
def __iter__(self):
yield input, target
and using the : torch.utils.data.DataLoader to wrap this dataset:
dataloader = DataLoader(train_dataset)
The CPU memory usage increases until the process is killed.
Does anyone ever encountered this or have an idea of how to solve this ? I tried to use the garbage collector of python but no difference?
The error raised from torch/utils/data/dataloader.py line 1256
raise RuntimeError(
RuntimeError: DataLoader worker (pid(s) 144359) exited unexpectedly
Important remark is that when the training happens in cpu, I do not have this memory leak issue, I only get this when training on gpu, and it’s not the gpu memory that gets full, it is ram.
Thank you all in advance!
I checked Memory increasing with IterableDataset and DataLoader memory usage keeps increasing but idea given in both.