I am using DataLoader to load my training data.
According to the document, we can set
num_workers to set the number of subprocess to speed up the loading process.
However, when I use it like this:
dataloader = DataLoader(dataset, batch_size=args.batch_size, shuffle=True, drop_last=False, num_workers=10, collate_fn=dataset.collate_fn)
I found the memory usage keep growing, which is not happening when I set
Is this supposed behavior ? I have limited memory resources, so I don’t want the memory usage keeps growing.
P.S. My code is running on the GPU, every time I move a batch of data from cpu to gpu.