Memory leak while iterating over dataloader

1

In an ideal case I prefer enumerating over dataloader with a for loop.
But for a particular implementation, I need multiple batches within one iteration in a sequence.
To do this I use, next(iter(dataloader)) to get as many batches as I need within one iteration.

The issue is, this causes linear increase in RAM over iterations and epochs. Somehow, the memory is not cleared after every iteration.

Any way I can avoid using iter and get multiple batches within one iteration?

2

store the iterator, don’t create it everytime

3

Sorry for the confusion. I am storing the iterator and using next() in every iteration to get the batch.

Both lead to memory leak.

4

Found the problem which was causing memory leak. The code I was using also uses itertools cycle() to iterate over another dataloader which was causing the memory leak.

Using iter(dataloader) does not cause any memory leak.

5

Why cycle cause memory leak???

I also experience same issue…

6

Hi Gullal,

I have a similar problem, what do you mean with itertools was using cycle() ? Like the normal iter() function? So how did you solve it?

I am using iter(dataloader) and it causes a memory leak