I currently load data with
toarch.load because they are saved as a pickle. Pickle can only load everything once into the memory. Then, I work on this data using a data loader and save this data in
data = torch.load('clean_data') test_loader = dataloader(data, batch_size=32, shuffle=True) result =  for img, label in test_loader: # do somehting result.append([img.cpu(), label.cpu()]) torch.save(result)
So I double use the cpu memory and I dont have so much available.
I want to save CPU memory. Does anybody has any idea? For example, how to load dynamically this pickle file? Or should I focus on saving the pickle file dynamically?
I am grateful for any help. Thanks