I want to find a data structure to save intermediate results. Numpy and Pickle is not an option because I cannot load the data in chunks.
The data is images, but saving .jpeg etc… is a loss of information.
How can I save these intermediate results and which enables me to load them memory efficient with a data loader?
I don’t quite understand the “cannot load the data in chunks” part, but given that I guess torch.save is not an option.
You could use a lossless compression format such as PNG.
I think that: Using instead of pickle or torch.save/torch.load. It is better to use tf.records, e.g. from the PyTorch DALI plugin.
What do you think about that?
Yes, DALI is certainly an option and compatible with PyTorch. I don’t know enough about tf.records to say if it’s needed or not.
Well, I think that tf.records are able to store the tuples (images, tuples). And load it batch-wise on the GPU. So that the RAM is not heavily used.
At the moment I use torch.save and torch.load, which has the drawback just loads files at once and if the files very huge, my RAM is too small.
I havnt seen anything comparable in the pytorch lib.