After code running for a long time, my dataloader just freezes. It seems like all subprocesses in dataloader hangs up and the main process just wait for dataloading.
I use Dataloader like this:
train_data_loader = DataLoader(train_data, batch_size=64,
shuffle=True, num_workers=10)
train_data is a data generation method.
The CPU usage:
The Memory usage:
You can see that after a several circles of model training and testing, the cpu and memory usage goes up and then goes down. But at last, when testing, the cpu usage goes 1/10, where i think the dataloader of testing just hangs up.
What happens of multiprocessing? There is no error until the work was killed.