I currently load data with toarch.load
because they are saved as a pickle. Pickle can only load everything once into the memory. Then, I work on this data using a data loader and save this data in list
.
data = torch.load('clean_data')
test_loader = dataloader(data, batch_size=32, shuffle=True)
result = []
for img, label in test_loader:
# do somehting
result.append([img.cpu(), label.cpu()])
torch.save(result)
So I double use the cpu memory and I dont have so much available.
I want to save CPU memory. Does anybody has any idea? For example, how to load dynamically this pickle file? Or should I focus on saving the pickle file dynamically?
I am grateful for any help. Thanks